- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether frozen, target-conditioned short-context drafters can retain their decoding speedups at much longer contexts without additional training. The method combines a bounded drafter attention cache with rotary indices assigned modulo the drafter’s implementation limit, while retaining full-context target features and verification against the complete prefix. The analysis gives conditions for common-shift RoPE invariance and acceptance bounds based on target truncation and drafter fidelity gaps. Experiments report that two DSpark drafters retain roughly their short-context decode-phase speedups through 2M tokens on Kimi-K3, with additional windowing results on Qwen through 512k and MiniMax through 1M. Window and sink ablations, cross-workload evaluations, and separate Qwen diagnostics examine the method’s behavior.
Critical strengths
- The frozen short-context drafters can accelerate decoding at million-token context lengths. With windowing and re-basing, both frozen DSpark drafters achieve decode-phase speedups over target-only decoding at 2M tokens (Table 1). Additional windowing gains on Qwen through 512k and MiniMax through 1M support the usefulness of bounded drafting beyond Kimi-K3 and DSpark (Section 6.3).
- The experiments clarify how computational cost and acceptance affect decoding speed. Section 6.2 distinguishes costly drafting despite substantial acceptance in unwindowed RadixArk from the acceptance collapse in windowed-only drafters beyond the rotary table’s limit. I find reporting emitted tokens per round alongside speedup useful for separating these failure modes. The verification-kernel comparison further shows why similar acceptance need not imply similar speedup (Appendix E.1).
Critical weaknesses
W1. The practical trade-offs of the positional design remain unclear.
-
I find Table 1 useful for showing that the proposed implementation works, but it is insufficient to establish the benefit of its positional choice over other bounds-safe implementations. Section 3 motivates the internal comparisons, yet the windowed-only arm in Section 6.2 fails beyond its rotary-table limit, as Section 4.2 anticipates and Appendix E.2 confirms. A comparison with bounds-safe true-index RoPE at the same frequencies and YaRN settings, using an extended table or safely computed rotations, would distinguish the benefit of modulo assignment from repairing invalid table access.
-
I would prioritize a matched comparison with StreamingLLM-style cache-relative indexing to evaluate the trade-off discussed in Sections 3 and 4.3. Re-basing avoids repeatedly rotating cached keys, but Section 4.3 acknowledges that it changes relative offsets across band boundaries and can affect subsequent outputs through cached states. Holding the frozen drafter, target, window, sink budget, and verification rule fixed would allow comparison of acceptance and runtime, including re-rotation costs and positions before, across, and sufficiently far after a boundary. I understand the authors do not claim superiority over cache-relative indexing, but this comparison would clarify what they gain and lose by that choice.
W2. The feature measurements do not establish the proposed explanation of long-context drafting.
-
I am not convinced that the feature replacement experiment supports the explanation offered at the end of Section 6.5. The introduction and Section 4.1 motivate short-context drafting through access to full-context target features, and Section 6.5 uses the feature replacement result to explain how a short-context drafter serves a 2M-token target. However, changing the drafter's output does not establish that the full-context features improve agreement with the full target. Does agreement improve under the same paired feature replacement? Without this comparison, I would limit the conclusion to the observation that the features affect the output.
-
I also think the scope of the fidelity measurements should be stated consistently. Section 4.1 refers to deployed fidelity, whereas Section 6.5 reports a fresh drafter pass that characterizes the checkpoint rather than the cached system. I recognize that the introduction and Section 7 explicitly avoid claiming that these diagnostics explain the deployment speedups; the wording in Section 4.1 should reflect that limitation as well.
Questions for authors
See weaknesses
4 comments
W2
W2: The benefit of full-context features depends on the task
We ran the requested paired comparison. Full-context features improve agreement with the full target when the task needs information from outside the drafter's window, and have almost no effect otherwise.
Here, "full-context features" are the target hidden states received by the drafter. The drafter itself always attends only to the last tokens, but the target computes these hidden states over the entire prefix, so they can carry distant information. On Qwen2.5-7B-1M with its EAGLE-3 drafter, we ran the same drafter twice at each position with identical tokens, positions, and window. The only difference was whether the target features were computed over the entire prefix or only the last tokens. We compared both conditions against the full target at 32k and 128k, with from 1,024 to 4,096.
- PG19 text completion: at the sampled continuation positions, recent context is generally sufficient, so full features add little (≈0 MAL at 32k, +0.03 to +0.05 at 128k).
- Needle-in-haystack (single-1, single-2, single-3): the needle lies outside the drafter's window and can reach the drafter only through the target's features.
| Needle-in-haystack (answer tokens) | Agreement: full / truncated | MAL: full / truncated |
|---|---|---|
| single-1 | 65–69% / 7–10% | 1.80–1.86 / 1.09–1.13 |
| single-2 | 72–78% / 5–7% | 1.96–2.07 / 1.05–1.08 |
| single-3 | 46–52% / 11–13% | 1.54–1.60 / 1.11–1.15 |
Full features raise MAL by 0.42–1.02 on answer tokens, across both context lengths and all window sizes. Therefore, they do not merely change the drafter's output; they improve its agreement with the full target.
The conclusion is task-dependent. Full-context features have little effect on locally predictable text completion, but they play a critical role in retrieval tasks such as needle-in-haystack: because the needle is outside the drafter's window, the target features are the only path by which that information reaches the drafter. We will revise Section 6.5 to state this narrower conclusion rather than using the diagnostic to explain the deployment speedups.
This experiment uses fresh drafter passes at selected positions, not the cached deployment trajectory. We will therefore replace "deployed fidelity" in Section 4.1 with the more accurate term "checkpoint-level diagnostic."
Revised version of the paper uploaded
Thank you again for your review and for engaging with our responses. We have uploaded a revised PDF that addresses the concerns raised by all reviewers, including the changes we promised in our replies to you. To make it easy to compare with the submitted version, all new or changed text is in red; it will be set in black in the camera-ready version.
We hope the revised version resolves the points you raised, and if so, we would appreciate it if you would reconsider your score. Should any question or concern remain, please let us know; we would be glad to look into it during the rest of the discussion period and try to address it. We appreciate the time you have put into reviewing our work.
Dear reviewer 4ZMr,
we are following up before the discussion period closes on Nov 18. We would be grateful to know whether our reply addresses your concerns. If it does, we would appreciate it if you would consider updating your score.
Thank you for your time and consideration.
W1
We sincerely thank the reviewer for the careful and constructive feedback. Your comments helped us [W1] clarify the practical trade-offs of our positional design and [W2] better explain the role and scope of full-context features. If you feel that our responses and planned revisions address your concerns, we would be grateful if you would consider them in your final score.
W1: Practical trade-offs of the positional design
Matched comparison at 2M. On Kimi-K3 with the Inferact drafter at 2M tokens (twice the drafter's 1,048,576-row RoPE table), we held the target, frozen drafter, K=3, W = 4,096, prompts, and verification fixed, and varied only the position scheme: extended RoPE (true positions, table extended to 2,097,152 rows), cache-relative (StreamingLLM-style positions 0,…,W−1, re-applied every step), re-basing (paper; positions modulo the table length), fixed re-basing (defined below), and window-only.
All schemes that keep distances correct tie within the ≈0.6 MAL resolution of n = 5; only window-only collapses, because it reads beyond the finite table.
The boundary. To isolate the interval in which the window straddles the table end, we used 1,046,528-token prompts and generated 8,192 tokens across it (MAL per segment):
The fix. As these results show, plain re-basing has a real, although rare, failure: it drops to 1.46 MAL while the window straddles the boundary, which affects W of every P generated tokens (about 0.4%). We found a simple, low-cost fix that removes it, and we will update the paper to use this corrected version of re-basing as the method. For now, we call it fixed re-basing. It works exactly like plain re-basing until the next query would reach P; at that point, it shifts all W cached keys together, once:
Kj←R(−δ)Kj(j=1,…,W),offset←offset+δ,
where δ=P−W, which moves the window to positions 0,…,W−1. Because every key receives the same shift, all relative distances remain exact and no position passes the table end. This happens once per about P−W tokens, adding ≈0.4% rotary work, ≈1% drafter time, and no extra memory. It removes the dip (3.47 MAL across the boundary vs. 3.45 for the control), and outside the crossing interval it is identical to the submitted method, so the paper's existing results remain valid.
Cost, not acceptance, distinguishes the schemes. Extended RoPE needs a table that grows with context; cache-relative re-rotates all W keys every step (≈10% drafter time overhead in this implementation); fixed re-basing needs neither. We will use extended RoPE and cache-relative as baselines in place of window-only, and present fixed re-basing as a low-memory, bounds-safe choice, not an acceptance improvement.
Matched comparison on Qwen3.5. For tighter statistics, we repeated the boundary test on Qwen3.5-35B-A3B with its frozen EAGLE-3 drafter (n = 20, 248k context, all settings fixed across schemes), setting P = 8,192 so that the generation crosses a boundary at step 1,024.
Fixed re-basing and cache-relative match true positions in every segment (paired differences on identical greedy streams: +0.000±0.004 and +0.001±0.003, vs. −0.082±0.050 for plain re-basing). The ≈2% overhead comes from our unfused prototype and the deliberately frequent boundary (P=2W); on Kimi-K3 it is ≈1% vs. ≈10% for cache-relative. We do not claim fixed re-basing is universally faster; its advantage is avoiding both a growing table and per-step re-rotation. We will add this comparison to the paper.