acceptodds
Under review as a conference paper at ICLR 2027

Efficient EAGLE-3 Training via Structure-Aware Execution

48 views

Abstract

Speculative decoding speeds up large language model inference using a lightweight drafter to propose multiple candidate tokens which the target model checks in parallel. EAGLE-3 is a recent draft-model architecture for speculative decoding. The data used to train drafters have heterogeneous supervision ratios. That is, the fraction of valid token positions with direct supervision varies across samples and datasets. Existing EAGLE-3 training frameworks still execute recurrent computations that cannot affect the training loss and recompute token-side projections for inputs shared by different recurrent paths, incurring avoidable computation and memory costs. We propose structure-aware execution based on two observations: some recurrent updates cannot affect a supervised prediction, and different computations can repeatedly project the same token-side input. These observations lead to three execution changes. First, we keep only recurrent computations needed by selected losses at the current or later training-time test (TTT) steps, while retaining their same-path predecessors and the full public key/value (K/V) context required by the retained queries. Second, we design fused attention to process these potentially non-contiguous queries without storing public-only outputs and log-sum-exp (LSE) values for a later merge. Third, we compute and reuse a shared token-side partial for each source position while separately projecting each computation's hidden state, avoiding repeated token-side projection work. On CNN and ShareGPT, our method achieves a 1.85–4.34 end-to-end training speedup over four existing training frameworks. Numerical checks show close agreement with SpecForge, and the trained drafters maintain comparable draft acceptance on held-out test examples.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, EAGLE-3, Drafter Training, Training Systems, GPU Optimization, Structured Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T7ZbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper optimizes the training efficiency of EAGLE-3’s training-time test (TTT). First, it removes computations on non-model-generated content (e.g., system prompts and queries) that do not contribute to the loss at different TTT steps (C1). Second, it designs a corresponding attention kernel for the remaining computations (C2). Third, it reuses the projection of the same token embedding across different TTT steps (C3).

Critical strengths

The problem is accurately identified. TTT, as a special form of training, indeed contains a large amount of redundant computation that can be eliminated, and the authors provide an effective optimization for this problem.

Critical weaknesses
  1. The method is designed for the specific draft model of EAGLE-3 with TTT, so its scope is somewhat narrow.
  2. The proposed method is currently evaluated only with batch size 1. Whether it can be readily adapted to large-batch training remains unclear.
Questions for authors
  1. Can the method be adapted to large-batch training? If so, could the authors provide some empirical results on batched training? If not, what are difficulties in doing so?
  2. Although a TTT length of 7 with full-sequence rollout follows the official EAGLE-3 release, in practice it is also reasonable to select only a subset of rollouts (e.g., anchored EAGLE-3 training in TorchSpec, PR #186). How do the authors view the difference between their method and anchored EAGLE-3 training? Since this work is very recent, an empirical comparison is not required; a discussion of the authors’ view is sufficient.
  3. Regarding C3, I do not fully understand why reusing embedding projection can reduce backward latency by more than 20% (while the forward latency is roughly the same). Could the authors explain this?
3 comments
AuthorsOpenReview ↗

W1. Scope: EAGLE-3 with TTT and Q1. Large-batch training.

W1. Scope: EAGLE-3 with TTT.

Our method applies to single-layer drafters trained with a multi-step unroll. At each step, each position uses its own state from the previous step. HASS, EAGLE-3, EAGLE 3.1 and EAGLE-4 share this structure. Our method therefore applies to them. EAGLE-1 and EAGLE-2 use a single training pass. They can use the first-step part of C1: build K/V for the full sequence and compute queries only at supervised positions.


Q1. Large-batch training.

Yes. Our method supports larger batches and uses less memory per GPU than the baseline.

  • What "batch size 1" means. It is the micro-batch size on each GPU. We used the standard training settings. The official EAGLE-3 settings use a micro-batch of 1 with gradient accumulation. The SpecForge EAGLE-3 recipes also use batch size 1.
  • Micro-batch > 1 on one GPU. We added support for this. We join the samples in each batch without padding. We checked that the loss and gradients match the SpecForge path with padding.
  • Results. We used one RTX 6000 Ada (48 GB), ShareGPT and Qwen3-8B. Each run trains on the same 2000 samples for two epochs. We time the second epoch. Memory is the peak allocated memory within a training step.
Micro-batchSpecForge memorySpecForge samples/sOurs memoryOurs samples/s
117.6 GiB2.3012.7 GiB3.74
235.4 GiB2.2120.3 GiB4.53
4out of memory–31.3 GiB4.76
  • Lower memory allows larger batches. Our method uses 28% less memory than SpecForge at micro-batch 1 and 43% less at micro-batch 2. It does not create padding. It also keeps only rows that can affect a loss. On this 48 GB GPU, our method fits micro-batch 4. The baseline fits at most micro-batch 2. For our method, throughput is 21% higher at micro-batch 2 and 27% higher at micro-batch 4 than at micro-batch 1. For the baseline, throughput is 4% lower at micro-batch 2 than at micro-batch 1. Both methods run out of memory at micro-batch 8 on this GPU.

  • Data parallelism across GPUs. Each GPU trains on its own samples. The GPUs communicate only through the wrapper's parameter all-gather and gradient reduce-scatter. C1–C3 therefore need no change. We checked the gradient passed to the optimizer on two GPUs. We compared it with the mean of the two GPUs' own gradients. The relative L2 error was 0.2% in BF16, using that mean as the reference. This check also covered a sample with no supervised token on one GPU. The table below uses the same samples and timing rules as above. We used FSDP, as in the paper. The machine has PCIe but no NVLink.

GPUsMicro-batch per GPUSpecForge samples/sOurs samples/s
213.354.65
223.546.07
24out of memory7.34
415.557.31
426.289.96
44out of memory11.99
  • What the multi-GPU results mean. For both methods, twice as many GPUs give less than twice the throughput on this machine. At micro-batch 1, SpecForge reaches 60% of ideal scaling on four GPUs. Our method reaches 49%. Here, ideal scaling means four times the one-GPU throughput. Our speedup over SpecForge falls from 1.63× on one GPU to 1.39× on two GPUs and 1.32× on four GPUs. We think this is due to PCIe communication at each step. It takes about the same time per step for both methods. Our step is shorter, so communication takes a larger share of its time. A larger micro-batch shares this cost across more samples. Our method fits a larger micro-batch. At micro-batch 4, it reaches 11.99 samples/s on four GPUs. This is 1.91× the best baseline result of 6.28 samples/s. Our method also reaches 7.34 samples/s on two GPUs, which is faster than the baseline on four GPUs.
AuthorsOpenReview ↗

Q2. Relation to Anchored EAGLE-3 Training (TorchSpec PR #186) and Q3. Why C3 Reduces Backward Time More Than Forward Time

Q2. Relation to anchored EAGLE-3 training (TorchSpec PR #186).

Thank you for pointing us to this work. PR #186 was merged on September 1, 2026, shortly before the submission deadline. The contribution included code but no paper. We did not know about it when we submitted our paper. We will cite it in the revision. We think the two methods can work together.

  • Which losses are used and how they are computed. Anchored training changes the training objective. It samples a fixed number N of anchor rollouts for each sample. It trains only on their losses. The loss set differs from that of dense EAGLE-3 training. The loss weights across samples also differ. Our method keeps the selected losses and their normalization the same. It only changes how they are computed. Section 5.3 checks this against SpecForge with numerical tests.
  • A shared observation. Both methods use the same fact about a one-layer drafter. The step-1 public K/V can be built for the full sequence. Queries only need to be computed where they are used.
  • The two can be combined. C1's loss-reachability rule (Eq. 1) works for any selected loss set. Each loss has a start position and a TTT step. RjR_j contains the start positions that still have a selected loss at step jj or later. For the anchored loss set, C1 gives the same execution as anchored training. It also skips trailing steps with unsupervised targets. C2 and C3 can then be added. C3 reuses the most work when anchors are next to each other. N such anchors unrolled for kk steps read only N+k−1N+k-1 distinct source positions, rather than NkNk. The reuse factor is close to kk. In the PR's results, adjacent anchors match the acceptance of dense training: both are 0.78. Scattered anchors have an acceptance of 0.47.

Q3. Why C3 reduces backward time much more than forward time.

Thank you for this question. We used Nsight Systems to profile the backward pass with C3 off and on. C1 was on and C2 was off in both runs. C3 saves backward work in two ways:

  1. Repeated backward work for input embeddings. Without C3, each TTT step looks up the embeddings of its shifted input tokens. With C3, we do this once when we build the token-side tables.
    • The optimizer does not update the drafter's input embedding. All our runs and the baseline use the same SpecForge training stack. FSDP (use_orig_params=True) puts this embedding and the trainable weights in the same flat parameter. As a result, autograd still computes the embedding gradient.
    • The backward pass of nn.Embedding uses a dense gradient buffer. Each lookup fills a full vocabulary-size buffer (151,936 × 4,096) with zeros and adds the gradients to it. It does this even when only a few tokens are looked up. A forward lookup only reads the rows it needs. This explains why backward has this extra cost.
    • C3 does this backward work once per unroll, rather than once per TTT step.
  2. Token-side Q/K/V projection gradients. C3 computes the token-side part of the Q/K/V weight and input gradients once per source position. Without C3, it computes this part once per use of that position. This saves projection-gradient GEMM work.

C3 also adds backward work. It casts values, gathers from the shared tables and adds up gradients from all uses of each shared entry. This extra work reduces the net savings.

If our responses address your concerns, we hope you will consider raising your score.

I thank the authors for the detailed response and the additional experiments on larger micro-batches, multi-GPU training, and the C3 backward analysis, which address my questions. I am raising my score to 3.

The method is technically solid, and the reported training speedups and memory savings are convincing. My remaining reservation concerns scope: the optimizations are specific to EAGLE-style TTT unrolling, which other drafter families do not require, and the contribution reads more as an efficiency improvement to existing training frameworks than as a transferable research insight. I leave the judgment of whether this meets the bar for ICLR to the AC.

Reviewer_k7riOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies efficient execution of EAGLE-3 drafter training while preserving the existing supervision and training objective. It introduces three optimizations: pruning recurrent computations that cannot contribute to a selected loss, fusing attention for the retained queries, and reusing token-side projections across recurrent computations. Experiments report training speedups of 1.85×–4.34× and reductions in peak allocated memory relative to four training frameworks. Numerical checks and held-out evaluations further indicate comparable trained-drafter acceptance and inference throughput to SpecForge.

Critical strengths
  1. The paper identifies that selective supervision does not require executing every recurrent computation in EAGLE-3 training. The distinction between computations needed for future losses and positions needed to supply attention context is useful and requires more care than simply applying the supervision mask.
  2. The reported reductions in training time and memory are meaningful within the evaluated configurations. Comparing against four training frameworks and measuring complete offline training runs provides stronger evidence than isolated kernel benchmarks alone.
  3. I appreciate the numerical comparisons of outputs, losses, gradients, and optimizer updates, together with paired training runs and held-out evaluations. These checks support the claim that the execution changes preserve the intended training behavior and the resulting drafter’s utility.
  4. The component ablations help explain the contributions of recurrent pruning, attention fusion, and projection reuse. The experiments varying supervision density and layout also provide useful insight into when the optimizations are most effective.
Critical weaknesses
  1. The practical significance of the training bottleneck needs stronger justification. My main concern is how the reported savings translate into the overall cost of preparing and deploying an EAGLE-3 drafter. The paper demonstrates substantial relative improvements, but the evaluated offline training runs are comparatively short: for example, Table 1 reduces the SpecForge runtime from approximately 53.4 to 15.6 minutes on CNN and from 2.89 to 1.55 hours on ShareGPT. These savings are useful, particularly for repeated adaptation, but their importance depends on the frequency of retraining and the cost of other preparation stages. Section 5.1 explicitly excludes target-feature generation from the measurements. Including this cost and evaluating a representative adaptation workflow would help establish when training execution is a substantial practical bottleneck.
  2. The inference evaluation is useful but insufficiently characterized for assessing deployment performance. Table 2(b) does report acceptance and throughput, including a comparison with the initial EAGLE-3 checkpoint. However, the inference runtime, draft configuration, sampling settings, and generation-length protocol are not sufficiently specified. A target-only autoregressive baseline is also absent, making it difficult to assess the absolute inference acceleration in this setup. A more complete deployment evaluation would help connect this preservation result to the claimed practical value of faster training.
  3. Limited applicability beyond EAGLE-3. The proposed implementation specifically targets EAGLE-3’s recurrent training-time-test topology. The pruning and projection-reuse mechanisms rely on dependencies and repeated inputs arising from this particular unrolling structure. Meanwhile, recent speculative decoding methods explore substantially different drafter architectures, including block-parallel generation in DFlash and a parallel backbone with a lightweight sequential module in DSpark. It is unclear which components of the proposed approach would remain applicable to these architectures and whether comparable training savings could be achieved. Given the rapid development of speculative decoding, a clearer characterization of the transferable principles, supported by an extension or analysis beyond EAGLE-3, would strengthen the broader significance of the contribution.
Questions for authors

Please check the weakness part above.

4 comments

W1. Practical significance of the training bottleneck.

Table 1 presents one concrete instantiation of our method, using a limited number of training samples and a single epoch. Practical drafter-training recipes use larger datasets and more epochs, and agentic reasoning workloads add longer sequences. The absolute training times in Table 1 therefore reflect this limited experimental scale. Below, we quantify the costs under longer training schedules and report results on long agent traces.

  • More epochs and more training data. Our Table 1 setup uses one epoch over 20,000 ShareGPT samples or 2,000 CNN samples, corresponding to 20,000 and 2,000 training steps, respectively. By comparison, the official EAGLE-3 training script uses 40 epochs, and the SpecForge EAGLE-3 example recipe uses 10 epochs. At a fixed batch size, larger datasets and more epochs require more training updates, increasing the absolute cost of preparing a drafter.
  • Cost of target-feature generation. We measured it for ShareGPT. Feature generation runs once. Training repeats in every epoch.
  • Total cost. The table adds feature-generation and training costs for ShareGPT, in GPU-hours. Total speedup is SpecForge's total cost divided by our total cost.
EpochsFeature generationSpecForge trainingOur trainingSpecForge totalOur totalTotal speedup
1 (Table 1)1.932.891.554.823.481.39×
10 (SpecForge recipe)1.9329.116.231.0318.131.71×
40 (official EAGLE-3 recipe)1.93116.963.6118.8365.531.81×
  • What the table shows. At 10 and 40 epochs, training accounts for 94% and 98% of SpecForge's combined feature-generation and training cost, respectively. Our method saves 12.9 and 53.3 GPU-hours per drafter in these two settings.
  • Repeated adaptation. This cost is paid again for each task and each target model. HedgeSpec trains task-specific EAGLE-3 drafters for summarization and text-to-SQL. Online Speculative Decoding and Aurora adapt drafters to queries received during use.
  • Longer sequences in agentic reasoning. Multi-step reasoning and repeated tool interactions produce longer training sequences, increasing the computation per sample. DRAgent contains agent traces of up to 12,288 tokens. Tool results and prompts make up most of each trace and are not supervised. Only 6.75% of the tokens are supervised. On 100 fixed DRAgent samples, our method reduces the mean step time from 1,078 ms to 288 ms. This is a 3.75× speedup. Peak allocated memory falls from 25.7 GiB to 12.5 GiB. The baseline here is the same code with C1–C3 turned off.
AuthorsOpenReview ↗

Larger batches

Larger batches.

Our method supports larger batches and uses less memory per GPU than the baseline.

  • What "batch size 1" means. It is the micro-batch size on each GPU. We used the standard training settings. The official EAGLE-3 settings use a micro-batch of 1 with gradient accumulation. The SpecForge EAGLE-3 recipes also use batch size 1.
  • Micro-batch > 1 on one GPU. We added support for this. We join the samples in each batch without padding. We checked that the loss and gradients match the SpecForge path with padding.
  • Results. We used one RTX 6000 Ada (48 GB), ShareGPT and Qwen3-8B. Each run trains on the same 2000 samples for two epochs. We time the second epoch. Memory is the peak allocated memory within a training step.
Micro-batchSpecForge memorySpecForge samples/sOurs memoryOurs samples/s
117.6 GiB2.3012.7 GiB3.74
235.4 GiB2.2120.3 GiB4.53
4out of memory–31.3 GiB4.76
  • Lower memory allows larger batches. Our method uses 28% less memory than SpecForge at micro-batch 1 and 43% less at micro-batch 2. It does not create padding. It also keeps only rows that can affect a loss. On this 48 GB GPU, our method fits micro-batch 4. The baseline fits at most micro-batch 2. For our method, throughput is 21% higher at micro-batch 2 and 27% higher at micro-batch 4 than at micro-batch 1. For the baseline, throughput is 4% lower at micro-batch 2 than at micro-batch 1. Both methods run out of memory at micro-batch 8 on this GPU.

  • Data parallelism across GPUs. Each GPU trains on its own samples. The GPUs communicate only through the wrapper's parameter all-gather and gradient reduce-scatter. C1–C3 therefore need no change. We checked the gradient passed to the optimizer on two GPUs. We compared it with the mean of the two GPUs' own gradients. The relative L2 error was 0.2% in BF16, using that mean as the reference. This check also covered a sample with no supervised token on one GPU. The table below uses the same samples and timing rules as above. We used FSDP, as in the paper. The machine has PCIe but no NVLink.

GPUsMicro-batch per GPUSpecForge samples/sOurs samples/s
213.354.65
223.546.07
24out of memory7.34
415.557.31
426.289.96
44out of memory11.99
  • What the multi-GPU results mean. For both methods, twice as many GPUs give less than twice the throughput on this machine. At micro-batch 1, SpecForge reaches 60% of ideal scaling on four GPUs. Our method reaches 49%. Here, ideal scaling means four times the one-GPU throughput. Our speedup over SpecForge falls from 1.63× on one GPU to 1.39× on two GPUs and 1.32× on four GPUs. We think this is due to PCIe communication at each step. It takes about the same time per step for both methods. Our step is shorter, so communication takes a larger share of its time. A larger micro-batch shares this cost across more samples. Our method fits a larger micro-batch. At micro-batch 4, it reaches 11.99 samples/s on four GPUs. This is 1.91× the best baseline result of 6.28 samples/s. Our method also reaches 7.34 samples/s on two GPUs, which is faster than the baseline on four GPUs.
AuthorsOpenReview ↗

W2. Inference tests and W3. Use with other drafters.

W2. Inference tests.

  • What Table 2(b) tests. We check whether a drafter trained with our method is as good as one trained with SpecForge. We do not claim a new inference speedup.
  • Settings.
ItemSetting
Inference engineSGLang 0.5.14, BF16, FlashInfer attention backend
HardwareOne RTX 6000 Ada (48 GB)
Draft configurationEAGLE-3 with speculative-num-steps 6, speculative-eagle-topk 10 and speculative-num-draft-tokens 32
SamplingGreedy decoding (temperature 0)
Generation lengthAt most 1024 new tokens per request
Inputs500 ShareGPT test conversations and 200 DRAgent test prefixes
OtherContext length 16,384. Three warm-up requests run before timing.
ThroughputGenerated tokens divided by the total request time
  • Target model alone. We also ran the target model without a drafter. We used the same settings.
ShareGPT (tokens/s)vs. target-onlyDRAgent (tokens/s)vs. target-only
Target-only53.381.00×40.541.00×
Vanilla EAGLE-3126.922.38×44.071.09×
SpecForge138.682.60×51.241.26×
Our method139.312.61×51.581.27×
  • What the results mean. DRAgent inputs are long. They have 1,445 to 11,523 tokens. The median is 7,027. Each request generates 1024 tokens. The request time includes processing the input. The vanilla drafter also has lower acceptance on DRAgent. It has 0.74 accepted drafts per verification on DRAgent and 2.47 on ShareGPT. On DRAgent, the vanilla drafter reaches 1.09× the throughput of the target model alone. Training on task data raises this to 1.27×. Such training is especially useful for agent workloads. Our method makes it cheaper.

W3. Use with other drafters.

  • Main idea. A position can provide attention context for other positions. It can also affect the training loss. We keep all positions needed for context. We skip work only when it cannot affect any selected loss. For each source position, we compute terms that depend only on its token once. We then reuse them across steps. These ideas also apply to other drafters.

  • Drafters with the same training design. Our method applies to drafters that have one layer and reuse it over several steps within each training pass. At each step, each position uses its own state from the previous step. HASS, EAGLE-3, EAGLE 3.1 and EAGLE-4 share this design. We also tested our code with the EAGLE 3.1 drafter settings. In an FP32 test on a small model, our loss and gradients match those from full computation. The relative error is below 3e-7.

  • EAGLE-1 and EAGLE-2. They train in one pass. They can use the first step of C1. This builds K/V for the full sequence and computes queries only at supervised positions.

  • DFlash and DSpark. They train in one pass. They sample anchor positions. They then predict a block of tokens after each anchor in parallel. We checked their public training code in SpecForge. The same idea can save work in two places.

    First, some rows have zero loss weight. The anchor row in each DFlash block is one example. Other rows still need these rows as attention keys. But their logits do not affect the loss. We can keep these rows in the backbone and skip their output-head work.

    Second, context positions at or after the last sampled anchor are never used as attention keys. We can skip their K/V computation.

    Joining samples without padding also works with different drafter types. We added this to support larger batches. We have not implemented these changes for DFlash or DSpark.

If our responses address your concerns, we hope you will consider raising your score.

Thank you for the response and additional experiments. Given that drafter training in Speculative Decoding typically involves far fewer trainable parameters than large-scale pretraining or RL, the urgency of optimizing it relative to inference efficiency remains insufficiently justified. The suggested DFlash and DSpark extensions differ from the core recurrent optimizations and have not been implemented, leaving the generalizability of the proposed algorithm unverified. Therefore, I will maintain my score.

Reviewer_whSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on accelerating EAGLE-3 drafter training. The authors observe that only part of the full sequence receives direct supervision, while current training frameworks still perform some redundant computations that do not affect the loss but increase training time. The optimizations of this paper focus on EAGLE-3 drafter training.

The first optimization, C1, reduces redundant computation in training-time test (TTT). It skips recurrent updates that cannot affect the current or later selected losses, while keeping the required predecessors and full public K/V. The second optimization, C2, fuses public and private attention in one forward kernel. It avoids writing and reading intermediate public-only outputs and LSE values, while keeping the final output and LSE for backward. The third optimization, C3, reuses the token-side Q/K/V projection partials for the same source position within one TTT unroll. The hidden-side projections are still computed separately.

With these three optimizations, the authors compare training costs with AngelSpec, SpecForge, TorchSpec, and Speculators at the same precision on CNN and ShareGPT. They report 1.85×–4.34× training speedup and a 29.05%–54.42% reduction in peak PyTorch allocated memory. These results use an offline training recipe and exclude target-feature generation time.

Critical strengths
  1. There are training speedups and memory savings under the tested settings. The ablation in Tab. 3 shows that C1 is the most effective component. Removing unnecessary TTT computation provides the main gains. At the same precision, the speedups on CNN are 3.4×–4.34×, with memory reduced by about 33%–54%. On ShareGPT, the speedups are about 1.8×–2.15×, with memory reduced by about 29%–49%. These results use Qwen3-8B.
  2. Tab. 2 compares numerical results and trained-drafter quality, which look close. Compared with SpecForge on DRAgent and ShareGPT, the L2 errors are small, and the relative differences in MAT and throughput are about 1.06% or less.
  3. The ablation study in Tab. 3 also shows that C1, C2, and C3 are effective. C1 accounts for more than 50% of the total step-time savings: 87.8% on CNN and 65.3% on ShareGPT. C2 and C3 contribute less.
  4. This work may help the development of the EAGLE series, such as EAGLE-4 [1].
  5. I think the C1/C2/C3 optimizations should not depend on the training backend and may still be effective with backends such as Megatron or FSDP2.

[1] https://github.com/eagle-hq/EAGLE-4/blob/main/EAGLE4_Paper.pdf

Critical weaknesses
  1. Regarding the contribution, the paper mainly improves the training speed of EAGLE-3 with TTT. It does achieve speedups over SpecForge and other frameworks. I think it would be useful to contribute these changes directly to SpecForge so the community can benefit, together with a technical report. However, to the best of my knowledge, the research ideas that readers can take from this work and apply elsewhere may be limited from an algorithm research perspective.

  2. The reasoning behind the optimizations is not very clear. In training or operator optimization, I think profiling should first identify the main costs, followed by optimizations targeting those costs. The paper reads as methods first, followed by gains. As a reader, I want to know why the authors prioritize the costs addressed by C1/C2/C3. What fraction of the total baseline training time do these parts take? What are the theoretical lower and upper limits of the possible improvements? Are the current C1/C2/C3 designs the best choices, and why?

    Would the profile be the same for Qwen3-8B and other models, such as a 35B model? I understand that this may be beyond the authors' available hardware. As a reader, I would like to see this complete optimization process. Although the results show that C1 provides the main gains, this does not explain how these costs were identified before the optimizations.

  3. It is not clear whether the contributions of C2/C3 are enough. For example, C2 mainly uses kernel fusion to reduce memory traffic. It gives about 1.06× speedup over the AngelSpec reference adapted to the same retained queries. Apart from the technical implementation itself, the additional ideas that C2/C3 provide to readers may be limited.

    From an optimization perspective, I also do not understand why C2/C3 were designed for further optimization if profiling does not show these parts to be the main training bottlenecks.

  4. Regarding Figure 4, I am familiar with training where all consecutive tokens in the final answer receive a loss. Figure 4 only gives a loss to position 5. Why not truncate the sequence at position 5? What roles do positions 6, 7, and 8 play? Showing 24→6 with this single-target example makes it difficult for me to understand what is saved when training on an entire final answer. Or are the authors trying to illustrate losses at separate positions across multiple turns, for example, with another loss at position 9, which would make the savings at positions 6–8 meaningful?

Questions for authors
  1. Could the authors provide a detailed profile of the baseline, showing how much training time is spent on the computations targeted by C1/C2/C3 and why these optimizations were prioritized?
  2. As discussed in the weaknesses, the connection between the motivation and the optimization designs, especially C2/C3, is not very clear. Profiling should probably come before the C1/C2/C3 optimizations.
  3. Why do the training frameworks in Tab. 1 still have these redundant computations? Which existing optimizations were enabled in the comparison? I am concerned about whether these baselines are strong enough and would like further clarification.
  4. Please clarify the possible misunderstanding of Fig. 4 discussed in the weaknesses.
  5. The paper focuses on accelerating EAGLE-3 drafter training. Its direct scope is training EAGLE-style drafters with a recurrent TTT structure. For parallel drafters without the same recurrent structure, such as DFlash, C1 does not directly apply. Could the authors explain which structural conditions these optimizations require and which ideas can apply beyond the current EAGLE-3 implementation?
7 comments
AuthorsOpenReview ↗

Q1, Q2, W2 and W3: Part 1. Profile of the baseline

Q1, Q2, W2 and W3: Part 1. Profile of the baseline

The table below shows the time spent in SpecForge and in each configuration of our method. We used ShareGPT and Qwen3-8B.

  • How to read the table. Times are in ms per optimizer step. Whole step, Forward and Backward are the full-run results in Table 3(b) of the paper. Embedding, Projection, Attention and MLP are module times. Each includes the forward and backward work of that module.
ConfigurationWhole stepForwardBackwardEmbeddingProjectionAttentionMLPOptimizer
SpecForge510.4201.4249.837.2120.783.4101.659.3
C1353.7127.9165.537.066.846.453.460.4
C1+C2313.5110.6143.538.066.912.653.559.5
C1+C3304.8126.2118.31.558.646.453.560.4
C1+C2+C3270.4109.4101.61.558.712.453.559.3
  • How much of the baseline step these modules take. Embedding, Projection, Attention and MLP take most of the baseline step time. These are the modules that C1–C3 change.
  • C1 lowers Projection, Attention and MLP. A TTT computation is one position at one TTT step. These modules run for every TTT computation. C1 skips the TTT computations that cannot affect any selected loss. This lowers the time spent in all three modules. These savings explain most of the drop in step time. Both the forward and backward passes become faster. Embedding time stays about the same.
  • C2 lowers only Attention. C2 computes the shared-prefix part and the per-query part of attention in one kernel pass. It stores no partial results. This lowers Attention time. Embedding, Projection and MLP stay about the same. The same pattern holds when C3 is on.
  • C3 lowers Embedding and Projection. Without C3, each TTT step looks up the token embeddings again. It also repeats the token-side projection for the same source position. With C3, both are done once, when the token-side tables are built. Embedding time falls sharply. Projection time also falls. Attention and MLP stay about the same. Almost all of the gain comes from the backward pass. Forward time changes little.
  • What C1–C3 do not change. Optimizer time stays about the same across all configurations. It remains a large part of the step time of our full method. The module times do not cover all forward and backward work. The rest is mainly time spent on the output head, the loss and data preparation.
AuthorsOpenReview ↗

Q1, Q2, W2 and W3: Part 2. Why we chose C1–C3

Q1, Q2, W2 and W3: Part 2. Why we chose C1–C3

  • Profiling is one way to find an optimization, but not the only one. A profile shows where the time goes. It does not show which work is unnecessary. We started from a different question: which computations in EAGLE-3 training are unused or repeated? We respectfully think that this is as valid a starting point as profiling first. It also finds things that a profile cannot find.
  • What a profile cannot show. A profiler such as Nsight Systems reports the time of each kernel. It cannot tell whether a kernel's result can reach a loss. It also cannot tell whether the same result was already computed. In the baseline profile, the recurrent steps look like normal Transformer work. Nothing in it shows that most TTT computations on CNN cannot affect any loss (C1). It does not show that the public attention result is stored and then read again for the merge (C2). It also does not show that the token-side projection is repeated for the same source position at every step (C3). We find these by checking the structure of the computation. We check which positions feed a loss, which keys the queries share, and which inputs stay the same across steps.
  • Ranking modules by time is too narrow. We can rank modules by time and tune the one that takes the most time. This makes that module faster. It does not remove work that should not run at all. C1–C3 remove such work. The profile above agrees with our analysis. We will add it to the paper. The profile checks the designs. The three designs came from our analysis of unnecessary work.
  • Why C1. Only some positions have a loss. A TTT computation that cannot reach any selected loss is unused work. On ShareGPT, which we used for our profile, many TTT computations are of this kind. C1 skips them. It keeps the first-step K/V for the full sequence because the kept rows attend to it.
  • Why C2 and C3. After C1, the recurrent work that is left is attention, projection and MLP on the kept rows. The MLP works on each row separately and has no repeated work. Attention and projection each have one kind of repeated work. Split attention first stores public-only results and merges them later. C2 computes both parts in one kernel pass. Projection repeats the token-side work for the same source position at every step. Each step also looks up the token embeddings again. C3 does both once.
  • What they add. In Table 3(b), adding C2 and C3 to C1 further lowers the step time on both CNN and ShareGPT.
  • When C1 has nothing to skip. When every position is supervised, C1 removes almost nothing. In Table 4, our method is still faster than SpecForge in this case. This gain does not come from skipping work.
  • Memory. Adding C3 to C1 lowers peak allocated memory on both CNN and ShareGPT. Using less memory allows larger batches on each GPU.
  • Gain from fusing attention. C2 gives a small gain over an AngelSpec-style attention that already uses the queries kept by C1. This comparison measures only the gain from computing the two attention parts in one kernel pass.
  • Are these the best designs? We do not claim that they are the best. Each part removes one kind of unused or repeated work. The costs that we know remain are the optimizer step and the gathers and gradient sums that C3 adds.
  • Where C2 and C3 may apply in other work. C2 applies to softmax attention where all queries share one set of keys. Each query also has its own keys. This structure is not specific to EAGLE-3. Hydragen (Juravsky et al., 2024) uses it for inference when many sequences share a prefix. This includes prefixes shared in a tree. We cite Hydragen in the related-work appendix. It computes attention over the shared prefix and over each private suffix separately. It then merges the two results. Hydragen targets inference. C2 targets training. It computes both parts in one kernel pass and stores no partial results. It also has a matching backward pass. It can therefore be used in training methods that attend to shared keys and private keys. One example is Online Draft Co-Training, which we also cite there. C3 applies to any layer whose input joins a fixed per-token input with a state that changes over steps. Examples are other drafters trained with a multi-step unroll and iterative refinement models with a fixed condition. The fixed input is projected once. The result is reused at every step.
AuthorsOpenReview ↗

Q3. Baselines and Q4 and W4. Figure 4

Q3. Baselines

  • The baselines are the latest training frameworks. SpecForge, AngelSpec, TorchSpec and Speculators are open-source frameworks for EAGLE-3 training. All are in active development. They come from the SGLang project, Tencent, the TorchSpec team and the vLLM project. For each framework, we used the latest commit of its main branch when we started the experiments. The commits are listed in the appendix. To our knowledge, these four frameworks are the state of the art in drafter training systems.
  • Why the frameworks still do this work. We think the reason is as follows. These frameworks run the TTT steps on the full sequence with dense tensors. This keeps the shapes fixed and lets them use standard attention kernels. Skipping rows needs three things together: a query set with gaps, K/V for the full sequence, and an attention kernel for this shape. Standard kernels do not support this. TorchSpec also supports anchored EAGLE-3 training. It also computes queries only where they are needed.
  • Settings shared by all frameworks. Each framework runs its own EAGLE-3 trainer at the commit listed in the appendix. All runs use BF16, the same batch size and the same TTT length. They use the same cached target features and the same supervision masks.
  • Optimizations enabled in each framework.
FrameworkEnabled
SpecForgeFlexAttention backend. Output head and loss only at selected positions.
AngelSpecFlexAttention backend with split public and private attention. Gradient checkpointing.
TorchSpecFlexAttention backend. Gradient checkpointing.
SpeculatorsIts own EAGLE-3 trainer. Gradient checkpointing is off.
  • Strength of the baselines. The table shows the optimizations that these frameworks provide. The baselines already use them. The SpecForge baseline already limits the output head and the loss to selected positions. For C2, the appendix also compares against an AngelSpec-style attention that uses the same kept queries.

Q4 and W4. Figure 4

  • Figure 4 is a schematic. It shows only which TTT computations C1 can skip. We put the supervised position in the middle to keep the figure small. The positions after the supervised position do not affect this loss. We agree that this is confusing.
  • Real sequences are already truncated. In our training data, each sequence ends with the supervised answer. There is nothing after it to cut off. Truncation therefore cannot remove any work.
  • Unsupervised positions cannot be removed. The positions before the answer have no loss, but the answer attends to them. They must stay as context. C1 keeps their first-step K/V and skips only their recurrent work. For example, consider a long sequence with a short supervised answer at the end. Full computation runs every position at every TTT step. C1 runs far fewer TTT computations. The same holds for the user turns between supervised spans in multi-turn data.
  • Change to Figure 4. We will move the supervised position to the end of the sequence. We will also change the caption. The computation counts for full computation and C1 will stay the same.

Comments on the paper

1 comment
AuthorsOpenReview ↗

GENERAL RESPONSE TO ALL REVIEWERS

We thank the reviewers for their careful reading and helpful comments. They noted the following strengths:

  • Finding and removing unnecessary training work. Reviewers noted that the paper finds and removes unnecessary TTT computation (k7ri, T7Zb). k7ri also found it useful to separate work needed for losses from positions needed as attention context.
  • Training speedups and memory savings. Reviewers noted the gains in the tested settings (whSH, k7ri, T7Zb).
  • Comparisons across training frameworks. k7ri valued the comparisons across four training frameworks and the timing of complete offline training runs. These tests provide evidence beyond individual kernel timings.
  • Careful checks of training behavior and drafter quality. Reviewers valued the numerical checks and tests of the trained drafters (whSH, k7ri). These checks support the claim that our method keeps the intended training behavior and gives similar inference performance.
  • Clear evidence of what each component adds. whSH and k7ri noted that the ablations show the gains from C1, C2, and C3. C1 gives the largest gain, and C2 and C3 provide further savings.
  • Tests showing when the method helps most. k7ri valued the tests that vary how many tokens are supervised and where they appear. These tests help explain when the optimizations are most effective.
  • Potential use in other EAGLE models and training backends. whSH noted that the work may help later EAGLE models, such as EAGLE-4. whSH also suggested that the optimizations may work with other training backends.

We responded to the main concerns as follows:

  1. Why we chose C1–C3 and what each adds (whSH, T7Zb). We added a profile showing the time spent in each module. We explained how each design removes unused or repeated work. We also explained the extra gains from C2 and C3 after C1, and why C3 mainly reduces backward time.

  2. Why faster training matters in practice (k7ri). We included the cost of target-feature generation. We explained why the savings matter more when training runs for more epochs or is repeated for different tasks and target models. We also reported results on long agent traces.

  3. Larger batches, more GPUs, and larger target models (T7Zb, whSH). We added tests showing that lower memory use allows larger micro-batches and higher throughput. These tests include FSDP training on multiple GPUs. We also reported a similar training speedup with Qwen3-32B. We explained the limits of multi-GPU scaling in our setup.

  4. Where the method applies (all reviewers). We explained the conditions each optimization needs. We reported a numerical check with EAGLE 3.1 settings. We also discussed how the ideas may apply to other drafter designs and made clear which changes we have not implemented.

  5. Baselines and inference tests (whSH, k7ri). We listed the baseline settings and the optimizations used by each framework. We also provided the inference settings and results for the target model alone. These details help explain our tests of drafter quality and inference performance.

  6. Anchored training and Figure 4 (T7Zb, whSH). We explained that anchored training selects which losses to use. Our method keeps the selected losses and their normalization the same. It changes how they are computed. We discussed how the two methods can work together. We also explained why context positions must stay even when their recurrent work can be skipped. We described our planned changes to Figure 4.

Our replies to each reviewer give the details and supporting results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.