acceptodds
Under review as a conference paper at ICLR 2027

Long-Context Drafting Does Not Require Long-Context Drafters

Abstract

Speculative decoding pairs a cheap drafter with parallel verification against the target. At ultra-long context it stops paying, because a drafter that attends over the full prefix inherits the target's context-dependent cost, so the speedup it was meant to provide first erodes and then reverses. The standard response is to build a long-context drafter, by training or fine-tuning one, or by using a multi-token-prediction head shipped inside the target checkpoint. We argue this is unnecessary, because the drafters in use are target-conditioned, reading hidden states the target computed over the entire prefix, so distant context already reaches them without their own attention, and verification corrects what they miss. We therefore freeze released short-context drafters and run them far past their training length under two inference-time mechanisms, one bounding the attention cache they see, which fixes their cost, and one re-basing their rotary position indices into the trained range. No weight is touched and the only tunable saturates early, so one vLLM-side configuration spans 4k to two million tokens, costing at most of speedup at short length. We prove when a common index shift leaves the drafter's computation exactly unchanged, and decompose acceptance into two measurable gaps with sharp bounds. Whatever speedup a frozen drafter delivers at short context, it retains out to two million tokens, across three target–drafter pairs and five corpora. The claim concerns the flatness rather than the multiple, and removing either mechanism eliminates it. The gain is conditional on the speculative cost per emitted token staying below the matched target-only cost. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, long-context inference, rotary position embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4ZMrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether frozen, target-conditioned short-context drafters can retain their decoding speedups at much longer contexts without additional training. The method combines a bounded drafter attention cache with rotary indices assigned modulo the drafter’s implementation limit, while retaining full-context target features and verification against the complete prefix. The analysis gives conditions for common-shift RoPE invariance and acceptance bounds based on target truncation and drafter fidelity gaps. Experiments report that two DSpark drafters retain roughly their short-context decode-phase speedups through 2M tokens on Kimi-K3, with additional windowing results on Qwen through 512k and MiniMax through 1M. Window and sink ablations, cross-workload evaluations, and separate Qwen diagnostics examine the method’s behavior.

Critical strengths
  1. The frozen short-context drafters can accelerate decoding at million-token context lengths. With windowing and re-basing, both frozen DSpark drafters achieve decode-phase speedups over target-only decoding at 2M tokens (Table 1). Additional windowing gains on Qwen through 512k and MiniMax through 1M support the usefulness of bounded drafting beyond Kimi-K3 and DSpark (Section 6.3).
  2. The experiments clarify how computational cost and acceptance affect decoding speed. Section 6.2 distinguishes costly drafting despite substantial acceptance in unwindowed RadixArk from the acceptance collapse in windowed-only drafters beyond the rotary table’s limit. I find reporting emitted tokens per round alongside speedup useful for separating these failure modes. The verification-kernel comparison further shows why similar acceptance need not imply similar speedup (Appendix E.1).
Critical weaknesses

W1. The practical trade-offs of the positional design remain unclear.

  • I find Table 1 useful for showing that the proposed implementation works, but it is insufficient to establish the benefit of its positional choice over other bounds-safe implementations. Section 3 motivates the internal comparisons, yet the windowed-only arm in Section 6.2 fails beyond its rotary-table limit, as Section 4.2 anticipates and Appendix E.2 confirms. A comparison with bounds-safe true-index RoPE at the same frequencies and YaRN settings, using an extended table or safely computed rotations, would distinguish the benefit of modulo assignment from repairing invalid table access.

  • I would prioritize a matched comparison with StreamingLLM-style cache-relative indexing to evaluate the trade-off discussed in Sections 3 and 4.3. Re-basing avoids repeatedly rotating cached keys, but Section 4.3 acknowledges that it changes relative offsets across band boundaries and can affect subsequent outputs through cached states. Holding the frozen drafter, target, window, sink budget, and verification rule fixed would allow comparison of acceptance and runtime, including re-rotation costs and positions before, across, and sufficiently far after a boundary. I understand the authors do not claim superiority over cache-relative indexing, but this comparison would clarify what they gain and lose by that choice.

W2. The feature measurements do not establish the proposed explanation of long-context drafting.

  • I am not convinced that the feature replacement experiment supports the explanation offered at the end of Section 6.5. The introduction and Section 4.1 motivate short-context drafting through access to full-context target features, and Section 6.5 uses the feature replacement result to explain how a short-context drafter serves a 2M-token target. However, changing the drafter's output does not establish that the full-context features improve agreement with the full target. Does agreement improve under the same paired feature replacement? Without this comparison, I would limit the conclusion to the observation that the features affect the output.

  • I also think the scope of the fidelity measurements should be stated consistently. Section 4.1 refers to deployed fidelity, whereas Section 6.5 reports a fresh drafter pass that characterizes the checkpoint rather than the cached system. I recognize that the introduction and Section 7 explicitly avoid claiming that these diagnostics explain the deployment speedups; the wording in Section 4.1 should reflect that limitation as well.

Questions for authors

See weaknesses

4 comments

We sincerely thank the reviewer for the careful and constructive feedback. Your comments helped us [W1] clarify the practical trade-offs of our positional design and [W2] better explain the role and scope of full-context features. If you feel that our responses and planned revisions address your concerns, we would be grateful if you would consider them in your final score.

W1: Practical trade-offs of the positional design

Matched comparison at 2M. On Kimi-K3 with the Inferact drafter at 2M tokens (twice the drafter's 1,048,576-row RoPE table), we held the target, frozen drafter, K=3K=3, WW = 4,096, prompts, and verification fixed, and varied only the position scheme: extended RoPE (true positions, table extended to 2,097,152 rows), cache-relative (StreamingLLM-style positions 0,…,W−10,\ldots,W-1, re-applied every step), re-basing (paper; positions modulo the table length), fixed re-basing (defined below), and window-only.

Position scheme at 2M (n = 5)MALSpeedupCompute overheadRoPE-table memory
Fixed re-basing3.222.49×≈1% drafter time1×
Re-basing (paper)3.352.59×≈0%1×
Extended RoPE3.152.44×≈0%2× at 2M; grows with context
Cache-relative3.072.36×≈10% drafter time1× + buffer
Window-only1.140.88×—invalid beyond table end

All schemes that keep distances correct tie within the ≈0.6 MAL resolution of n = 5; only window-only collapses, because it reads beyond the finite table.

The boundary. To isolate the interval in which the window straddles the table end, we used 1,046,528-token prompts and generated 8,192 tokens across it (MAL per segment):

Scheme (Inferact drafter)BeforeAcrossAfterDrafter-time overheadRoPE-table memory
Plain re-basing (paper)2.501.462.90≈0%1×
Fixed re-basing2.773.473.38≈1%1×
Extended RoPE (no crossing, control)2.653.453.36≈0%2× at 2M; grows with context
Cache-relative (127k; no table end)2.602.912.71≈10%1× + small buffer

The fix. As these results show, plain re-basing has a real, although rare, failure: it drops to 1.46 MAL while the window straddles the boundary, which affects WW of every PP generated tokens (about 0.4%). We found a simple, low-cost fix that removes it, and we will update the paper to use this corrected version of re-basing as the method. For now, we call it fixed re-basing. It works exactly like plain re-basing until the next query would reach PP; at that point, it shifts all WW cached keys together, once:

Kj←R(−δ) Kj(j=1,…,W),offset←offset+δ,K_j \leftarrow R(-\delta)\,K_j \quad (j = 1, \ldots, W), \qquad \text{offset} \leftarrow \text{offset} + \delta,

where δ=P−W\delta = P - W, which moves the window to positions 0,…,W−10, \ldots, W-1. Because every key receives the same shift, all relative distances remain exact and no position passes the table end. This happens once per about P−WP - W tokens, adding ≈0.4% rotary work, ≈1% drafter time, and no extra memory. It removes the dip (3.47 MAL across the boundary vs. 3.45 for the control), and outside the crossing interval it is identical to the submitted method, so the paper's existing results remain valid.

Cost, not acceptance, distinguishes the schemes. Extended RoPE needs a table that grows with context; cache-relative re-rotates all WW keys every step (≈10% drafter time overhead in this implementation); fixed re-basing needs neither. We will use extended RoPE and cache-relative as baselines in place of window-only, and present fixed re-basing as a low-memory, bounds-safe choice, not an acceptance improvement.

Matched comparison on Qwen3.5. For tighter statistics, we repeated the boundary test on Qwen3.5-35B-A3B with its frozen EAGLE-3 drafter (n = 20, 248k context, all settings fixed across schemes), setting PP = 8,192 so that the generation crosses a boundary at step 1,024.

Position scheme at 248kBeforeAcrossAfterDrafter-time overhead
True positions3.013.603.59≈0%
Plain re-basing3.013.423.59≈0%
Fixed re-basing3.013.603.59≈2%
Cache-relative3.013.603.59≈1%

Fixed re-basing and cache-relative match true positions in every segment (paired differences on identical greedy streams: +0.000±0.004+0.000\pm0.004 and +0.001±0.003+0.001\pm0.003, vs. −0.082±0.050-0.082\pm0.050 for plain re-basing). The ≈2% overhead comes from our unfused prototype and the deliberately frequent boundary (P=2WP=2W); on Kimi-K3 it is ≈1% vs. ≈10% for cache-relative. We do not claim fixed re-basing is universally faster; its advantage is avoiding both a growing table and per-step re-rotation. We will add this comparison to the paper.

W2: The benefit of full-context features depends on the task

We ran the requested paired comparison. Full-context features improve agreement with the full target when the task needs information from outside the drafter's window, and have almost no effect otherwise.

Here, "full-context features" are the target hidden states received by the drafter. The drafter itself always attends only to the last WW tokens, but the target computes these hidden states over the entire prefix, so they can carry distant information. On Qwen2.5-7B-1M with its EAGLE-3 drafter, we ran the same drafter twice at each position with identical tokens, positions, and window. The only difference was whether the target features were computed over the entire prefix or only the last WW tokens. We compared both conditions against the full target at 32k and 128k, with WW from 1,024 to 4,096.

  • PG19 text completion: at the sampled continuation positions, recent context is generally sufficient, so full features add little (≈0 MAL at 32k, +0.03 to +0.05 at 128k).
  • Needle-in-haystack (single-1, single-2, single-3): the needle lies outside the drafter's window and can reach the drafter only through the target's features.
Needle-in-haystack (answer tokens)Agreement: full / truncatedMAL: full / truncated
single-165–69% / 7–10%1.80–1.86 / 1.09–1.13
single-272–78% / 5–7%1.96–2.07 / 1.05–1.08
single-346–52% / 11–13%1.54–1.60 / 1.11–1.15

Full features raise MAL by 0.42–1.02 on answer tokens, across both context lengths and all window sizes. Therefore, they do not merely change the drafter's output; they improve its agreement with the full target.

The conclusion is task-dependent. Full-context features have little effect on locally predictable text completion, but they play a critical role in retrieval tasks such as needle-in-haystack: because the needle is outside the drafter's window, the target features are the only path by which that information reaches the drafter. We will revise Section 6.5 to state this narrower conclusion rather than using the diagnostic to explain the deployment speedups.

This experiment uses fresh drafter passes at selected positions, not the cached deployment trajectory. We will therefore replace "deployed fidelity" in Section 4.1 with the more accurate term "checkpoint-level diagnostic."

AuthorsOpenReview ↗

Revised version of the paper uploaded

Thank you again for your review and for engaging with our responses. We have uploaded a revised PDF that addresses the concerns raised by all reviewers, including the changes we promised in our replies to you. To make it easy to compare with the submitted version, all new or changed text is in red; it will be set in black in the camera-ready version.

We hope the revised version resolves the points you raised, and if so, we would appreciate it if you would reconsider your score. Should any question or concern remain, please let us know; we would be glad to look into it during the rest of the discussion period and try to address it. We appreciate the time you have put into reviewing our work.

Dear reviewer 4ZMr,

we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.

Thank you for your time and consideration.

Reviewer_eTgEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper proposes a training-free approach to enable long-context speculative decoding with short-context drafters. The key idea is to bound the drafter's attention cost using a windowed KV cache and address the rotary position extrapolation issue through RoPE index re-basing. The paper argues that target-conditioned drafters, such as EAGLE-style drafters, can still benefit from target hidden states even when their own attention is limited to a short window. Experiments on several target-drafter pairs demonstrate improved speedup retention at million-token contexts.

Critical strengths
  1. The paper identifies an important efficiency bottleneck of speculative decoding under ultra-long contexts. Reducing the drafter KV cost is a meaningful direction for long-context inference.
  2. The proposed windowing and RoPE re-basing mechanism is simple and does not require additional training or modifying model weights, which makes it potentially practical for existing speculative decoding systems.
Critical weaknesses
  1. The core idea of restricting draft-side attention to a local window has already been explored in prior speculative decoding and MTP-based approaches. The authors themselves identify Windowed-MTP as the closest work, the main difference of this paper is extending the idea from a native MTP head to an external target-conditioned drafter, together with an additional RoPE re-basing mechanism.

  2. Another important concern would be the unexplained token-level discrepancy between speculative and target-only decoding. The paper reports that the two arms do not produce a token-by-token identical stream on Kimi-K3, with divergence appearing as early as the first generated token. The authors attribute this to numerical nondeterminism caused by different decode batch shapes, but this explanation is not directly verified.

  3. The paper is also unnecessarily complicated in its presentation, and the role of the theoretical analysis in supporting the main empirical claim is unclear. The authors present the two-gap analysis as a “measured mechanism” and use it to argue that acceptance is drafter-limited rather than window-limited. However, the gap measurements are conducted under a different evaluation law and on different model pairs from the main speedup experiments. The paper itself explicitly states that “we do not claim the latter explains the former.”

Questions for authors

Q1: Could the authors clarify what fundamental algorithmic novelty is introduced beyond extending the idea to an external drafter and handling its RoPE range?

Q2: Could the authors provide controlled experiments to verify that the observed token-level discrepancy is indeed caused by numerical nondeterminism from different decode batch shapes?

Q3: Could the authors provide evidence under the same deployment setting that the two-gap analysis explains the observed acceptance behavior and speedup saturation?

9 comments

We sincerely thank the reviewer for the thoughtful and detailed feedback. Your comments helped us [Q1] better distinguish our contribution from prior windowed-drafting work; [Q2] identify the source of the reported token discrepancy using an independent reference implementation; and [Q3] clarify the scope of the two-gap analysis by measuring the proposal states reached by the deployed configuration. These additions strengthen both our empirical evidence and the precision of our claims. If you feel that our responses and planned revisions address your concerns, we would be grateful if you would consider them in your final score.

Q1: Novelty beyond local-window drafting

Our contribution is best summarized by the paper's title: train short, serve long. A frozen, external, target-conditioned drafter, trained independently and only on short sequences, keeps its speedup at contexts far beyond its training length and its own RoPE table, with no retraining. On Kimi-K3, a drafter trained at 4,096 tokens retains its speedup at 2M, about 500× its training length.

This requires two components, and neither alone is sufficient:

  • Windowing keeps the drafter's acceptance from collapsing as context grows.
  • re-basing keeps the drafter's positions valid beyond the end of its RoPE table. At 2M on Kimi-K3, window-only collapses to MAL, while windowing with re-basing keeps MAL flat and the speedup.

We do not claim windowed draft attention itself as novel; we credit MagicDec and Windowed-MTP for it.

How this differs from prior work. MagicDec drafts with the target itself, and Windowed-MTP windows an MTP head that is jointly trained and shipped by the model provider. Neither uses a frozen external drafter beyond its own position range. To our knowledge, off-the-shelf external drafters had not previously been shown to retain speedup at million-token scale, far beyond their training length.

Side-by-side comparison. To make the distinction concrete, we ran a new experiment on Qwen3.5-35B-A3B, which has both a native MTP head and a strong public EAGLE-3 drafter. The target supports 262k tokens; the single-layer EAGLE-3 drafter was trained on at most 8,192 tokens. We used K=3K = 3, WW = 4,096, greedy decoding, 8,192 generated tokens, PG19, and n=20n = 20.

Method at 248kSpeedup / MAL
EAGLE-3 as shipped1.09× / 1.47
Windowed EAGLE-3 (ours)3.05× / 3.50
Native MTP2.87× / 3.93
Windowed-MTP3.33× / 3.92
MagicDec (self-speculation)1.40× / 3.26

For the external short-context drafter, windowing prevents acceptance from collapsing (MAL 1.47 → 3.50) and recovers the speedup. For the native MTP head, acceptance is already high and unchanged by windowing; the window only makes each round cheaper. Windowed-MTP is faster here because Qwen3.5's provider-trained MTP head is a stronger drafter than the community EAGLE-3 head: the gap is already present at 8k (MAL 3.91 vs 3.54), where position handling plays no role. MagicDec keeps high acceptance but runs all 40 target layers per draft token, so it reaches only 1.40×.

In practice, our method lets the community reuse frozen EAGLE- or DSpark-style drafters for ultra-long generation without long-context retraining or a provider-shipped MTP head, which matters for many models similar to Kimi-K3 that ship none.

Q2: Token-level discrepancy

The divergence comes from the vLLM execution path, not from our method. vLLM's documentation (https://docs.vllm.ai/en/stable/features/speculative_decoding/) states that speculative decoding is “theoretically lossless up to the precision limits of hardware numerics” and identifies “Batch Size and Numerical Stability” as a source of output variation. Our speculative arm verifies K+1K+1 rows while target-only decodes one; different tensor shapes can select different kernels and reduction orders, so a near-tied greedy argmax can flip and the continuations then diverge.

To isolate the algorithm from engine numerics, we re-ran the Table 1 Kimi-K3 evaluation (8k to 2M) in an independent plain-PyTorch reference implementation with the same target, drafter, K=3K = 3, windowing, and re-basing. For every prompt, both speculative configurations were token-identical to target-only greedy decoding:

ContextnWindowed speedupUnwindowed speedupIdentical to target-only
8k52.62×2.62×10/10
32k52.13×2.09×10/10
128k52.07×1.33×10/10
512k52.05×0.90×10/10
1M52.24×0.78×10/10
2M52.37×0.80×10/10

All 60 speculative outputs match, up to 2M tokens, which rules out windowing or re-basing as the cause. We will replace the limitation statement in the paper with this result and note that the vLLM divergence reflects its documented batch-shape-dependent numerics, not a loss of correctness.

Q3: Role of the two-gap analysis

We re-measured the two-gap analysis on the same setting as the speedups. The analysis answers one specific question: when the drafter disagrees with the full target, is the error caused by limiting attention to WW tokens, or by the drafter itself? It explains acceptance behavior, not end-to-end speedup; speedup also depends on the costs of drafting and verification. We will make this scope explicit in the paper.

We used Qwen2.5-7B-1M + EAGLE-3, for which the paper also reports speedups (Table 2), so the analysis and the speedups now come from the same model and drafter. To measure the states the deployed drafter actually reaches, we used a Hugging Face replay of the deployed EAGLE-3 proposal graph, validated against vLLM with a chained-step logit error of 8.6×10−68.6 \times 10^{-6}. This is a validated replay of the deployed configuration, not instrumentation inside the live vLLM engine. We evaluated PG19 and RULER, with contexts from 32k to 256k and WW = 256 to 16,384. At each proposal state, we recorded:

  • aa: the token selected by the full target;
  • bb: the token selected by the target when restricted to the same WW-token window as the drafter;
  • cc: the token proposed by the windowed drafter.

From these tokens, we measure:

  • ρ=Pr⁡(a≠b)\rho = \Pr(a \neq b): the error caused by the window;
  • δ=Pr⁡(b≠c)\delta = \Pr(b \neq c): the remaining drafter error;
  • α=Pr⁡(a=c)\alpha = \Pr(a = c): drafter–target agreement.

The theory predicts

max⁡(0, 1−ρ−δ)  ≤  α  ≤  1−∣ρ−δ∣.\max(0,\ 1 - \rho - \delta) \;\leq\; \alpha \;\leq\; 1 - |\rho - \delta|.

All 84 workload–length–window combinations satisfy these bounds. Representative results are:

WorkloadContextWWindow error ρDrafter error δAgreement αMost common outcome (share of states)Predicted range for α
PG1932k1,0240.1790.6230.383Full and windowed targets agree; drafter differs (47.3%)[0.20, 0.56]
PG1932k4,0960.1320.6290.377Full and windowed targets agree; drafter differs (51.6%)[0.24, 0.50]
PG19128k1,0240.1730.6240.383Full and windowed targets agree; drafter differs (47.7%)[0.20, 0.55]
PG19128k4,0960.1230.6130.383Full and windowed targets agree; drafter differs (51.3%)[0.26, 0.51]
RULER needle answer 1128k4,0960.8760.9240.450Drafter and full target agree; windowed target differs (40.2%)[0, 0.95]
RULER needle answer 2128k4,0960.8680.8970.511Drafter and full target agree; windowed target differs (45.2%)[0, 0.97]

The PG19 result is straightforward. The window error is relatively small, while the remaining drafter error is large. Increasing WW from 256 to 16,384 reduces the window error by about threefold, but agreement with the full target stays nearly unchanged. Therefore, adding more local context does not improve acceptance: on ordinary text, the drafter itself is the bottleneck.

RULER needle answers show a different regime. Because the answer lies outside the local window, the window-restricted target often fails. However, the drafter still receives full-context features from the target and therefore frequently matches the full target when the window-restricted target does not: this is the most common outcome, in 40–45% of answer states. In this regime the theoretical bounds remain valid but are too wide to explain acceptance precisely.

We will therefore replace the broad “measured mechanism” wording with a more precise conclusion. The two-gap analysis is a diagnostic of acceptance: its bounds hold in every setting we measured; on ordinary text, it shows that acceptance saturates with WW because the drafter's own error dominates; and on retrieval, it shows that the drafter matches the full target where the windowed target cannot, so distant information reaches the drafter through the target's features. End-to-end speedup additionally depends on drafting and verification costs, which we report separately.

Thank you for the additional experiments. The results show that draft-side windowing and RoPE re-basing can help frozen external drafters operate at longer contexts. But I still think the manuscript needs a substantial revision to improve readability. For example , the authors could use the available space in the main method figure to illustrate the corrected re-basing procedure, plot key results to make comparisons easier to read, and use short \paragraph{} headings to highlight the main takeaways. If the revision could addresses readability concerns, I would raise my score to 2.

Reviewer_DaiqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an existing short-context drafter can still accelerate a target model at very long contexts, without further training. The approach is straightforward: keep only a small attention window for the drafter, reuse the target's context-aware features, and re-base the drafter's rotary positions when needed. The target still verifies against the full context. Experiments with DSpark and EAGLE-family drafters show that this can preserve useful decoding speedups as the context grows, including results at two million tokens.

Critical strengths
  1. I like the simplicity and practical motivation. A drafter that repeatedly reads the entire history can become too expensive to be useful, even if its predictions are reasonably good. Reusing full-context target features while limiting the drafter's own attention is a sensible way to address this. Windowed drafting has precedents, including MagicDec and Windowed-MTP, but adapting existing external drafters to this setting still seems worthwhile.

  2. The reported gains are meaningful, and there is more evidence than a single favorable example. For instance, the MiniMax/EAGLE-3.1 comparison improves from 1.75x to 2.62x target-only speed at 1M context, while the Kimi-K3/Inferact configuration retains a 2.25x speedup at 2M. I also appreciate the separation between unwindowed, windowed-only, and windowed plus re-basing configurations. The additional model pairs and EOS-stopped BEAM results make the practical claim more convincing.

Critical weaknesses
  1. My main concern is the lack of a direct comparison with other long-context speculative decoding methods. Appendix C explains why the authors keep the comparisons internal, but I do not find those reasons sufficient to omit external baselines altogether. Keeping the drafter fixed is useful for an ablation; it does not tell us how competitive the resulting system is. I am not asking for every baseline to run on Kimi-K3 at 2M. A comparison on a compatible smaller target, within the context range supported by both methods, would already be informative.

  2. The re-basing ablation does not fully separate the proposed idea from fixing an implementation failure. Appendix E.2 attributes the windowed-only collapse beyond Pmax to an unchecked out-of-range RoPE-table access. Recovering from that failure is useful, but I would also like to see a baseline that handles the lookup correctly, using extended or on-demand rotary factors while keeping the weights and frequencies fixed. A cache-relative position scheme would be another relevant comparison. The behavior around a modulo boundary also deserves a closer look, since the common-shift argument does not cover offsets across that boundary.

Questions for authors

See Weaknesses. A convincing comparison under aligned settings, together with clear answers to these checks, would make me comfortable raising my score. At present, these gaps keep my recommendation close to the acceptance boundary.

3 comments

We sincerely thank the reviewer for the detailed and constructive feedback. Your comments led us to add [W1] a matched comparison among external EAGLE-3 drafting, native MTP, Windowed-MTP, and MagicDec, and [W2] bounds-safe positional baselines, a direct boundary analysis, and fixed re-basing. These additions strengthen the empirical comparison and correct a rare boundary-crossing failure in the submitted method. If these results and revisions address your concerns, we would be grateful if you would consider them in your final score.

W1: External baselines

We ran the requested comparison on Qwen3.5-35B-A3B because it has both a native MTP head and a strong public EAGLE-3 drafter, allowing all baselines to be evaluated side by side. The target supports 262,144 tokens, while the single-layer drafter was trained on sequences of at most 8,192 tokens.

Setup: vLLM, K=3K=3, W=4,096W=4{,}096, greedy decoding, 8,192 generated tokens, PG19, and n=20n=20. Speedup is decode throughput over target-only on the same prompt (speedup / MAL):

ContextEAGLE-3 as shippedOurs (windowed EAGLE-3)Native MTP headWindowed-MTPMagicDec (self-spec)
8k1.65× / 2.352.49× / 3.542.61× / 3.922.57× / 3.910.73× / 3.17
128k1.15× / 1.572.80× / 3.442.79× / 3.933.10× / 3.931.13× / 3.24
248k1.09× / 1.473.05× / 3.502.87× / 3.933.33× / 3.921.40× / 3.26

Native MTP is strong when available, and our windowed EAGLE-3 matches or exceeds its speedup at 128k–248k. However, MTP heads are jointly trained and shipped by the model provider; Kimi-K3, for example, has none. External drafters can instead be trained by the community on a frozen target, as this EAGLE-3 head was. The table also shows that the unwindowed drafter degrades as context grows. Prior work usually handles long-context drafting through long-context training [LongSpec; DFlash; DSpark], while training-free methods stop near 128k and within the drafter's position range [TriForce; SpecExtend]. We show that a frozen short-context drafter can retain speedup at 2M, about 500× its training length, without retraining. Windowing prevents the external drafter's MAL from collapsing, whereas MTP acceptance remains unchanged (3.92–3.93 MAL) and only its round cost decreases. The approaches are therefore complementary.

We also compared against MagicDec, where the target drafts for itself through a StreamingLLM cache containing attention sinks and the 4,096 most recent tokens. Although its acceptance is high (MAL ≈3.3), each draft token runs all 40 target layers rather than one EAGLE-3 layer. At 248k, a round therefore costs 2.3 target decode steps, versus 1.15 for ours. MagicDec is slower than target-only at 8k (0.73×) and reaches only 1.13× at 128k and 1.40× at 248k.

We will add this comparison to the paper and revise Appendix C accordingly.

W2: Re-basing vs. a correct lookup, cache-relative positions, and the boundary

The window-only collapse beyond the table end is a lookup failure, so we compared re-basing with two bounds-safe alternatives and measured the boundary directly.

W2a. Bounds-safe position schemes. RoPE attention depends on relative distance (Eq. 2), so schemes that preserve all in-window distances should differ only in cost. We compared them on Kimi-K3 at 2M, twice the drafter's 1,048,576-row RoPE table:

  • Extended RoPE: the table is extended to 2,097,152 rows with the same weights and frequencies, so every key keeps its true position.
  • Cache-relative: keys are re-rotated to positions 0,…,W−10, \ldots, W-1 at every step.
  • Re-basing (paper): positions are taken modulo the table length.
Scheme (2M, n = 5)Inferact (MAL / speedup)
Re-basing (paper)3.35 / 2.59×
Fixed re-basing (see W2b)3.22 / 2.49×
Extended RoPE3.15 / 2.44×
Cache-relative3.07 / 2.36×
Window-only (no position handling)1.14 / 0.88×

The valid schemes differ by less than the ≈0.6 MAL resolution of n = 5; only window-only collapses. Thus, the benefit comes from windowing with valid positions, while the position schemes differ mainly in cost.

W2b. The boundary.

Plain re-basing fails only while the window (WW = 4,096 tokens) straddles the end of the table (PP = 1,048,576 rows). This affects W/PW/P = 0.39% of generated tokens; a request generating GG = 4,096 tokens crosses such an interval with probability (W+G)/P(W+G)/P = 0.78%. We measured the failure using 1,046,528-token prompts and 8,192 generated tokens:

Scheme (Inferact drafter)BeforeAcrossAfterDrafter-time overheadRoPE-table memory
Plain re-basing (paper)2.501.462.90≈0%1×
Fixed re-basing (ours)2.773.473.38≈1%1×
Extended RoPE (no crossing, control)2.653.453.36≈0%2× at 2M, grows with context
Cache-relative†2.602.912.71≈10%1× (+ small buffer)

MAL per segment. †Cache-relative uses the same protocol at 127k; it has no table boundary to cross.

As these results show, plain re-basing has a real, although rare, failure: it drops to 1.46 MAL while the window straddles the boundary. We found a simple, low-cost fix that removes it, and we will update the paper to use this corrected version of re-basing as the method. We call it fixed re-basing. It works exactly like plain re-basing until the next query would reach PP; at that point, it shifts all WW cached keys together, once:

Kj←R(−δ) Kj(j=1,…,W),offset←offset+δ,K_j \leftarrow R(-\delta)\,K_j \quad (j = 1, \ldots, W), \qquad \text{offset} \leftarrow \text{offset} + \delta,

where δ=P−W\delta = P - W, which moves the window to positions 0,…,W−10, \ldots, W-1. Because every key receives the same shift, all relative distances remain exact. This operation occurs once per approximately P−WP - W tokens, not once per crossing token, adding W/(P−W)≈W/(P-W) \approx 0.4% rotary work, approximately 1% drafter-time overhead, and no extra memory. It removes the dip: 3.47 MAL across the boundary, versus 3.45 for the extended-RoPE control that never crosses.

The alternatives trade computation for memory: cache-relative re-rotates all WW keys every step (≈10% drafter-time overhead), while extended RoPE doubles the table at 2M and grows with context. Fixed re-basing changes behavior only during the rare crossing interval; everywhere else it is identical to the submitted method, so the paper's existing results remain valid.

We will report extended RoPE and cache-relative as baselines in place of window-only, add the boundary measurement, and use fixed re-basing as the method.

AuthorsOpenReview ↗

Revised version of the paper uploaded

Thank you again for your review and for engaging with our responses. We have uploaded a revised PDF that addresses the concerns raised by all reviewers, including the changes we promised in our replies to you. To make it easy to compare with the submitted version, all new or changed text is in red; it will be set in black in the camera-ready version.

We hope the revised version resolves the points you raised, and if so, we would appreciate it if you would reconsider your score. Should any question or concern remain, please let us know; we would be glad to look into it during the rest of the discussion period and try to address it. We appreciate the time you have put into reviewing our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions to the paper

We thank all reviewers and the AC for their feedback. We have uploaded a revised PDF. To make it easy to compare with the submitted version, all new or changed text is in red; it will be set in black in the camera-ready version. The changes are summarized below.

1. Re-basing completed (Reviewer Daiq, W2, and Reviewer 4ZMr, W1; addressed in Section 4.3, Figure 2 and Corollary 1). The submitted version assigned indices modulo the drafter's table, which left offsets inexact across the table end. Re-basing now keeps the assigned indices growing and, when the next index would reach PmaxP_{max}, rotates the WW keys in the window back by Pmax−WP_{max}-W in one step. Every relative offset the drafter sees is then exact at every step, and Corollary 1 now has a proof. The extra rotary work is about 0.4% amortized. This is a completed version of the same method, not a new one: no setting in Table 1 crosses the table end, so its results are unchanged. The band-boundary bound in Appendix A.1, which the completed version no longer needs, is removed.

2. New experiments (added in a new Section 6.4 with Figure 3, and in a new Appendix J with Tables 14–19).

  • External baselines (Reviewer Daiq, W1; addressed in Appendix J.1, Table 14). On Qwen3.5-35B-A3B up to 248k, we compare EAGLE-3 as shipped, native MTP, Windowed-MTP and MagicDec. Windowing takes the same frozen EAGLE-3 head from 1.09× to 3.05× at 248k, above native MTP (2.87×) and MagicDec (1.40×).
  • Bounds-safe position schemes (Reviewer Daiq, W2, and Reviewer 4ZMr, W1; addressed in Appendix J.2, Table 15). On Kimi-K3 at 2M, re-basing, true positions from an extended table, and StreamingLLM-style cache-relative positions reach the same MAL (3.07–3.22), while window-only collapses to 1.14. Cache-relative positions cost about 10% more drafter time; re-basing costs about 1%.
  • Crossing the table end (Reviewer Daiq, W2, and Reviewer 4ZMr, W1; addressed in Appendix J.3, Table 16). Across the crossing, re-basing gives MAL 3.47, matching 3.45 for true positions, on Kimi-K3 and on Qwen3.5-35B-A3B.
  • Output equivalence (Reviewer eTgE, Q2; addressed in Appendix J.4, Table 17). In an independent plain-PyTorch implementation, speculative outputs from 8k to 2M are token-identical to target-only greedy decoding.
  • Feature swap with agreement (Reviewer 4ZMr, W2; addressed in Appendix J.5, Table 18). Under the same paired swap, full-context features improve agreement with the full target by 0.42–1.02 MAL on needle answers.
  • Gaps at reached states (Reviewer eTgE, Q3; addressed in Appendix J.6, Table 19). At the states the deployed drafter actually reaches, the bounds hold in all 84 settings, and acceptance is limited by the drafter's own error rather than the window.

3. Clarified scope (Reviewer 4ZMr, W2, and Reviewer eTgE, Q3; addressed in Sections 1, 4.1 and 7, and Appendices C and H). Section 4.1 now describes fidelity as a checkpoint-level diagnostic. The Introduction and Limitations state that the gap analysis explains acceptance, not end-to-end speedup. Appendix C notes the newly added baselines, and Appendix H is updated for the new results.

4. Readability (Reviewer eTgE, follow-up comment; addressed in the Introduction, Sections 4–6, and Figures 2 and 3). We itemized the contributions and added short paragraph headings in Sections 4–6 that state the main point of each paragraph. Figure 2 now illustrates re-basing at the table end, and Figure 3 plots the new comparisons.

We hope the revision makes the changes easy to review, and we would be glad to answer any further questions during the remaining discussion period.

AuthorsOpenReview ↗

Second revision

Dear Reviewers and Area Chair,

Following Reviewer eTgE's follow-up comment, we have uploaded a second revision that makes Section 6 (Experiments) easier to read. Changes from this round are shown in blue, and the first round's changes remain in red. All of them will be in black in the camera-ready version.

We would like to mention that most of the colored parts in the paper rewrite existing content to make it easier to follow. The method, experiments, and results are unchanged, and the only new material is what was added in the first revision to address the technical concerns of the reviewers.

Summary of the second revision:

  1. Section 6 rewritten. We rewrote the whole results section to make the results clearer and easier to follow.

  2. Figure 3 redrawn. The markers are larger and easier to tell apart, and a dotted line marks target-only decoding. Panel (b) now compares acceptance at 2M for the four ways of handling positions. The table-end crossing results remain in Table 16.

We thank all reviewers again for their time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.