- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SharpDraft, a training-free method for improving long-context speculative decoding. The authors attribute declining draft acceptance to attention-mass dilution as context length increases. SharpDraft addresses this by dynamically scaling attention queries according to the number of visible keys, requiring no parameter updates or modifications to the target model. Experiments on DFlash, PARD, and EAGLE 3.1 demonstrate improvements in acceptance length and end-to-end decoding speed.
Critical strengths
- SharpDraft requires no additional training and introduces negligible computational and memory overhead.
- The method improves decoding efficiency across multiple draft model families and benchmarks.
- The paper provides an analytical formulation of attention-mass dilution, supported by ablation studies.
Critical weaknesses
- Missing sliding-window attention baselines. Attention Drift has already demonstrated that sliding-window attention (SWA) can substantially improve EAGLE's acceptance length in long contexts. Moreover, recent official DFlash checkpoints also employ sliding-window attention, including models trained on 40K-token sequences. Since SharpDraft specifically addresses attention dilution caused by growing key counts, SWA is a particularly relevant baseline that deserves comparison and discussion.
- Limited evidence on long-context-trained drafters. The primary DFlash models were trained with sequences of only 3,072 tokens but evaluated on much longer generations. It remains unclear whether SharpDraft primarily mitigates training–inference length mismatch or addresses a more general attention-dilution problem.
- Modest gains over fixed scaling. Table 3 shows that fixed query scaling already achieves most of the improvement, with dynamic scaling providing only an additional ~1% MAL gain.
Questions for authors
- Comparison with SWA: How does SharpDraft compare with sliding-window drafting under the same evaluation settings? Are the two approaches complementary?
- Long-context training: Does SharpDraft remain effective when applied to drafters trained on longer sequences, particularly recent DFlash checkpoints trained with long-context data and sliding-window attention?
- Dynamic vs. fixed scaling: Given the small improvement over the best fixed scale, could the authors further clarify when dynamic scaling is preferable in practice?
4 comments
Author Response [2/2]
Weakness 2: Limited evidence on long-context-trained drafters. The primary DFlash models were trained with sequences of only 3,072 tokens but evaluated on much longer generations. It remains unclear whether SharpDraft primarily mitigates training–inference length mismatch or addresses a more general attention-dilution problem.
Question 2: Long-context training: Does SharpDraft remain effective when applied to drafters trained on longer sequences, particularly recent DFlash checkpoints trained with long-context data and sliding-window attention?
On the 40K-trained Qwen3.5-27B-DFlash, adding SharpDraft at a fixed 8K SWA window improves measured MAL and throughput on all four tasks (Section 4.4, Table 3). These comparisons include different continuations; their benefit should not be interpreted as an identical-output latency gain.
Training length and visible-key exposure are distinct: long sequences under SWA do not expose each query to the whole sequence. Comparing Query scaling at a fixed SWA window tests its effect under the same context-access constraint.
What the added evidence supports. The 40K-trained checkpoint results show that Query scaling can remain useful after long-context training. Separately, Appendix D.2 shows that uniform all-layer Query scaling at fixed states can reduce long-context draft–target mismatch.
This checkpoint comparison does not control for training length independently of model and attention configuration.
Weakness 3: Modest gains over fixed scaling.Table 3shows that fixed query scaling already achieves most of the improvement, with dynamic scaling providing only an additional ~1% MAL gain.
Question 3: Dynamic vs. fixed scaling: Given the small improvement over the best fixed scale, could the authors further clarify when dynamic scaling is preferable in practice?
Dynamic scaling is useful when one configuration must handle varied requests without selecting a new fixed scale for each workload. Its additional mean throughput gains over each task’s fastest tested fixed scale are 0.53–2.15%. Across 120 requests, it reaches at least 98% of each request’s best measured fixed-setting throughput on 97.50% of requests, compared with at most 65.80% for any single fixed scale (Figure 6(c)).
Comparison across tasks. Table 4 reports all five fixed scales on AIME-26, GPQA-D, LongGenBench Block Short, and LongWeave Paragraph Ordering 4K. Each task uses 30 requests per method, including repetitive and capped outputs.
Full fixed-scale sweep (Table 4). Cells show throughput in tokens/s (MAL). Bold marks the best overall result; underlining marks the best fixed result (ties included).
| Method | AIME-26 | GPQA-D | Block Short | Paragraph 4K |
|---|---|---|---|---|
| Frozen () | 91.33 (3.41) | 99.31 (3.25) | 165.59 (5.05) | 224.06 (6.84) |
| Fixed | 113.34 (3.86) | 128.78 (3.57) | 229.67 (6.87) | () |
| Fixed | 129.69 (4.17) | 153.81 (3.82) | 264.46 (7.79) | 232.02 (6.93) |
| Fixed | 136.83 (4.27) | 164.28 () | () | 226.26 (6.70) |
| Fixed | () | () | 270.15 (7.96) | 216.86 (6.40) |
| Fixed | 137.89 (4.23) | 164.43 (3.87) | 265.81 (7.78) | 206.91 (6.07) |
| SharpDraft | 139.28 (4.33) | 169.48 (4.02) | 277.32 (8.12) | 234.21 (7.03) |
The preferred fixed scale can also change with generation length. On GPQA-D, the highest mean cumulative MAL is obtained with at 4K, at 16K, and at 32K generated tokens. SharpDraft maintains top-32 attention mass near 0.76–0.77 over 2K–32K, while fixed-scale mass declines (Figure 6(a,b)). Each position averages only requests that reach it.
When to use dynamic scaling. The preferred constant changes across tasks, yet SharpDraft has the highest observed mean throughput on all four. The per-request comparison uses the best throughput among five measured fixed scales, selected retrospectively for each request. The observed coverage favors automatic adjustment for variable workloads; a stable workload can use a well-calibrated fixed scale.
The conclusion is limited to the evaluated greedy comparisons. The small mean timing differences do not establish statistical significance.
In summary, the same-window comparisons show that SharpDraft can complement SWA, including on the evaluated 40K-trained drafter. The task-dependent results clarify the benefit of access to distant context, while the fixed-scale analysis identifies varied workloads as a setting where dynamic scaling is useful.
Best regards,
Authors
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts and will maintain my Weak Accept score.
Dear reviewer CriU,
Thank you for your positive assessment and constructive feedback. We appreciate your time and thoughtful engagement throughout the discussion.
Best regards,
Authors
Author Response [1/2]
Dear Reviewer CriU,
Thank you for your thoughtful feedback. We have added SWA comparisons with DSpark-8B and DFlash2-8B and broadened the fixed-versus-dynamic analysis. The results clarify when full context and dynamic scaling are useful. We have revised the manuscript and highlighted the changes in blue.
SWA and SharpDraft can be combined: SWA selects the visible keys, and SharpDraft adjusts their attention weights. At a fixed 8K window, DFlash2-8B shows small observed throughput gains of 0.37–1.27% from adding scaling. The table includes unscaled Full and native-window controls, so gains from broader context access can be distinguished from scaling (
Section 4.4,Table 3). For DSpark-8B, SWA is implemented on its released full-attention checkpoint.Table R7. Throughput in tokens/s (MAL). Full uses full draft attention without calibration; SharpDraft adds Query scaling. The target verifier is unchanged, and all repetitive and capped outputs are included. The 8B arms agree in output tokens and termination. SharpDraft uses N0=4K for DSpark-8B and N0=3K for DFlash2-8B and the 40K-trained DFlash. The 40K-trained DFlash comparisons include different continuations.Task-dependent benefits. On DFlash2-8B, SharpDraft increases throughput over native SWA on Paragraph Ordering, while their throughput is similar on GPQA-D. This contrast is consistent with task-dependent reliance on distant context. On DSpark-8B, requests with larger matched-prefix acceptance gains from Full + SharpDraft over 2K SWA also tend to show larger full-generation MAL gains (r=0.91;
Figure 4).Combining SWA and SharpDraft. The fixed-window comparisons show that SharpDraft can improve throughput when combined with SWA. Window size trades off access to distant context against attention cost, while scaling adjusts the retained keys and saturates once the visible-key count stops growing. Jointly tuning window size and reference cardinality for each workload remains future work.
(Continued in the next comment)