- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper optimizes the training efficiency of EAGLE-3’s training-time test (TTT). First, it removes computations on non-model-generated content (e.g., system prompts and queries) that do not contribute to the loss at different TTT steps (C1). Second, it designs a corresponding attention kernel for the remaining computations (C2). Third, it reuses the projection of the same token embedding across different TTT steps (C3).
Critical strengths
The problem is accurately identified. TTT, as a special form of training, indeed contains a large amount of redundant computation that can be eliminated, and the authors provide an effective optimization for this problem.
Critical weaknesses
- The method is designed for the specific draft model of EAGLE-3 with TTT, so its scope is somewhat narrow.
- The proposed method is currently evaluated only with batch size 1. Whether it can be readily adapted to large-batch training remains unclear.
Questions for authors
- Can the method be adapted to large-batch training? If so, could the authors provide some empirical results on batched training? If not, what are difficulties in doing so?
- Although a TTT length of 7 with full-sequence rollout follows the official EAGLE-3 release, in practice it is also reasonable to select only a subset of rollouts (e.g., anchored EAGLE-3 training in TorchSpec, PR #186). How do the authors view the difference between their method and anchored EAGLE-3 training? Since this work is very recent, an empirical comparison is not required; a discussion of the authors’ view is sufficient.
- Regarding C3, I do not fully understand why reusing embedding projection can reduce backward latency by more than 20% (while the forward latency is roughly the same). Could the authors explain this?
3 comments
Q2. Relation to Anchored EAGLE-3 Training (TorchSpec PR #186) and Q3. Why C3 Reduces Backward Time More Than Forward Time
Q2. Relation to anchored EAGLE-3 training (TorchSpec PR #186).
Thank you for pointing us to this work. PR #186 was merged on September 1, 2026, shortly before the submission deadline. The contribution included code but no paper. We did not know about it when we submitted our paper. We will cite it in the revision. We think the two methods can work together.
- Which losses are used and how they are computed. Anchored training changes the training objective. It samples a fixed number N of anchor rollouts for each sample. It trains only on their losses. The loss set differs from that of dense EAGLE-3 training. The loss weights across samples also differ. Our method keeps the selected losses and their normalization the same. It only changes how they are computed. Section 5.3 checks this against SpecForge with numerical tests.
- A shared observation. Both methods use the same fact about a one-layer drafter. The step-1 public K/V can be built for the full sequence. Queries only need to be computed where they are used.
- The two can be combined. C1's loss-reachability rule (Eq. 1) works for any selected loss set. Each loss has a start position and a TTT step. contains the start positions that still have a selected loss at step or later. For the anchored loss set, C1 gives the same execution as anchored training. It also skips trailing steps with unsupervised targets. C2 and C3 can then be added. C3 reuses the most work when anchors are next to each other. N such anchors unrolled for steps read only distinct source positions, rather than . The reuse factor is close to . In the PR's results, adjacent anchors match the acceptance of dense training: both are 0.78. Scattered anchors have an acceptance of 0.47.
Q3. Why C3 reduces backward time much more than forward time.
Thank you for this question. We used Nsight Systems to profile the backward pass with C3 off and on. C1 was on and C2 was off in both runs. C3 saves backward work in two ways:
- Repeated backward work for input embeddings. Without C3, each TTT step looks up the embeddings of its shifted input tokens. With C3, we do this once when we build the token-side tables.
- The optimizer does not update the drafter's input embedding. All our runs and the baseline use the same SpecForge training stack. FSDP (
use_orig_params=True) puts this embedding and the trainable weights in the same flat parameter. As a result, autograd still computes the embedding gradient. - The backward pass of
nn.Embeddinguses a dense gradient buffer. Each lookup fills a full vocabulary-size buffer (151,936 × 4,096) with zeros and adds the gradients to it. It does this even when only a few tokens are looked up. A forward lookup only reads the rows it needs. This explains why backward has this extra cost. - C3 does this backward work once per unroll, rather than once per TTT step.
- The optimizer does not update the drafter's input embedding. All our runs and the baseline use the same SpecForge training stack. FSDP (
- Token-side Q/K/V projection gradients. C3 computes the token-side part of the Q/K/V weight and input gradients once per source position. Without C3, it computes this part once per use of that position. This saves projection-gradient GEMM work.
C3 also adds backward work. It casts values, gathers from the shared tables and adds up gradients from all uses of each shared entry. This extra work reduces the net savings.
If our responses address your concerns, we hope you will consider raising your score.
I thank the authors for the detailed response and the additional experiments on larger micro-batches, multi-GPU training, and the C3 backward analysis, which address my questions. I am raising my score to 3.
The method is technically solid, and the reported training speedups and memory savings are convincing. My remaining reservation concerns scope: the optimizations are specific to EAGLE-style TTT unrolling, which other drafter families do not require, and the contribution reads more as an efficiency improvement to existing training frameworks than as a transferable research insight. I leave the judgment of whether this meets the bar for ICLR to the AC.
W1. Scope: EAGLE-3 with TTT and Q1. Large-batch training.
W1. Scope: EAGLE-3 with TTT.
Our method applies to single-layer drafters trained with a multi-step unroll. At each step, each position uses its own state from the previous step. HASS, EAGLE-3, EAGLE 3.1 and EAGLE-4 share this structure. Our method therefore applies to them. EAGLE-1 and EAGLE-2 use a single training pass. They can use the first-step part of C1: build K/V for the full sequence and compute queries only at supervised positions.
Q1. Large-batch training.
Yes. Our method supports larger batches and uses less memory per GPU than the baseline.
Lower memory allows larger batches. Our method uses 28% less memory than SpecForge at micro-batch 1 and 43% less at micro-batch 2. It does not create padding. It also keeps only rows that can affect a loss. On this 48 GB GPU, our method fits micro-batch 4. The baseline fits at most micro-batch 2. For our method, throughput is 21% higher at micro-batch 2 and 27% higher at micro-batch 4 than at micro-batch 1. For the baseline, throughput is 4% lower at micro-batch 2 than at micro-batch 1. Both methods run out of memory at micro-batch 8 on this GPU.
Data parallelism across GPUs. Each GPU trains on its own samples. The GPUs communicate only through the wrapper's parameter all-gather and gradient reduce-scatter. C1–C3 therefore need no change. We checked the gradient passed to the optimizer on two GPUs. We compared it with the mean of the two GPUs' own gradients. The relative L2 error was 0.2% in BF16, using that mean as the reference. This check also covered a sample with no supervised token on one GPU. The table below uses the same samples and timing rules as above. We used FSDP, as in the paper. The machine has PCIe but no NVLink.