acceptodds
Under review as a conference paper at ICLR 2027

SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Abstract

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentrated on the highest-scoring Keys can decrease. We introduce SharpDraft, a training-free method that counteracts this effect through cardinality-aware Query scaling, without the computational overhead of online adaptation. Under explicit assumptions, we derive an exact top- mass correction and deploy a closed-form fixed-slope approximation. Across AIME-26, GPQA-Diamond, and LongGenBench Diary, SharpDraft achieves – geometric-mean end-to-end speedups over target-only autoregressive decoding when applied to DFlash, PARD, and EAGLE 3.1. With DFlash, it improves decoding speed without increasing measured peak allocated GPU memory and outperforms full-parameter and LoRA-based online adaptation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
speculative decoding, long-context inference, inference acceleration, attention calibration, query scaling, training-free adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CriUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SharpDraft, a training-free method for improving long-context speculative decoding. The authors attribute declining draft acceptance to attention-mass dilution as context length increases. SharpDraft addresses this by dynamically scaling attention queries according to the number of visible keys, requiring no parameter updates or modifications to the target model. Experiments on DFlash, PARD, and EAGLE 3.1 demonstrate improvements in acceptance length and end-to-end decoding speed.

Critical strengths
  1. SharpDraft requires no additional training and introduces negligible computational and memory overhead.
  2. The method improves decoding efficiency across multiple draft model families and benchmarks.
  3. The paper provides an analytical formulation of attention-mass dilution, supported by ablation studies.
Critical weaknesses
  1. Missing sliding-window attention baselines. Attention Drift has already demonstrated that sliding-window attention (SWA) can substantially improve EAGLE's acceptance length in long contexts. Moreover, recent official DFlash checkpoints also employ sliding-window attention, including models trained on 40K-token sequences. Since SharpDraft specifically addresses attention dilution caused by growing key counts, SWA is a particularly relevant baseline that deserves comparison and discussion.
  2. Limited evidence on long-context-trained drafters. The primary DFlash models were trained with sequences of only 3,072 tokens but evaluated on much longer generations. It remains unclear whether SharpDraft primarily mitigates training–inference length mismatch or addresses a more general attention-dilution problem.
  3. Modest gains over fixed scaling. Table 3 shows that fixed query scaling already achieves most of the improvement, with dynamic scaling providing only an additional ~1% MAL gain.
Questions for authors
  1. Comparison with SWA: How does SharpDraft compare with sliding-window drafting under the same evaluation settings? Are the two approaches complementary?
  2. Long-context training: Does SharpDraft remain effective when applied to drafters trained on longer sequences, particularly recent DFlash checkpoints trained with long-context data and sliding-window attention?
  3. Dynamic vs. fixed scaling: Given the small improvement over the best fixed scale, could the authors further clarify when dynamic scaling is preferable in practice?
4 comments
AuthorsOpenReview ↗

Author Response [1/2]

Dear Reviewer CriU,

Thank you for your thoughtful feedback. We have added SWA comparisons with DSpark-8B and DFlash2-8B and broadened the fixed-versus-dynamic analysis. The results clarify when full context and dynamic scaling are useful. We have revised the manuscript and highlighted the changes in blue.


Weakness 1: Missing sliding-window attention baselines. Attention Drift has already demonstrated that sliding-window attention (SWA) can substantially improve EAGLE's acceptance length in long contexts. Moreover, recent official DFlash checkpoints also employ sliding-window attention, including models trained on 40K-token sequences. Since SharpDraft specifically addresses attention dilution caused by growing key counts, SWA is a particularly relevant baseline that deserves comparison and discussion.

Question 1: Comparison with SWA: How does SharpDraft compare with sliding-window drafting under the same evaluation settings? Are the two approaches complementary?

SWA and SharpDraft can be combined: SWA selects the visible keys, and SharpDraft adjusts their attention weights. At a fixed 8K window, DFlash2-8B shows small observed throughput gains of 0.37–1.27% from adding scaling. The table includes unscaled Full and native-window controls, so gains from broader context access can be distinguished from scaling (Section 4.4, Table 3). For DSpark-8B, SWA is implemented on its released full-attention checkpoint.

DrafterMethodAIME-26GPQA-DBlock ShortParagraph 4K
DSpark-8BFull (Default)147.00 (4.25)170.00 (4.11)235.25 (6.62)220.48 (6.05)
DSpark-8BSWA (2K)150.87 (4.26)165.42 (3.97)163.76 (4.88)80.21 (2.26)
DSpark-8BSWA (8K)152.61 (4.32)172.25 (4.13)236.51 (6.65)225.41 (6.05)
DSpark-8BSWA (8K) + Ours153.33 (4.34)172.00 (4.13)236.83 (6.67)226.45 (6.07)
DSpark-8BFull + Ours152.34 (4.34)167.87 (4.14)230.58 (6.69)226.90 (6.07)
DFlash2-8BFull136.86 (3.97)170.11 (4.03)215.87 (6.17)126.71 (3.65)
DFlash2-8BSWA (2K, Default)140.69 (4.02)171.21 (4.02)176.63 (5.18)98.33 (2.76)
DFlash2-8BSWA (8K)140.18 (4.03)172.02 (4.06)221.43 (6.32)128.48 (3.65)
DFlash2-8BSWA (8K) + Ours141.84 (4.06)172.66 (4.07)224.12 (6.39)130.11 (3.70)
DFlash2-8BFull + Ours141.79 (4.06)173.04 (4.07)224.65 (6.39)130.45 (3.70)
DFlash-27B (40K-trained)Full50.41 (4.44)46.91 (5.07)50.63 (3.16)201.46 (12.96)
DFlash-27B (40K-trained)SWA (4K, Default)130.89 (7.85)100.09 (6.05)80.87 (4.95)183.95 (11.85)
DFlash-27B (40K-trained)SWA (8K)75.57 (5.04)75.28 (5.29)60.09 (3.77)195.59 (12.96)
DFlash-27B (40K-trained)SWA (8K) + Ours132.22 (7.98)97.95 (6.06)77.46 (4.96)205.80 (13.58)
DFlash-27B (40K-trained)Full + Ours131.11 (7.93)98.72 (6.03)79.40 (4.95)211.82 (13.58)

Table R7. Throughput in tokens/s (MAL). Full uses full draft attention without calibration; SharpDraft adds Query scaling. The target verifier is unchanged, and all repetitive and capped outputs are included. The 8B arms agree in output tokens and termination. SharpDraft uses N0=4KN_0=4\mathrm{K} for DSpark-8B and N0=3KN_0=3\mathrm{K} for DFlash2-8B and the 40K-trained DFlash. The 40K-trained DFlash comparisons include different continuations.

Task-dependent benefits. On DFlash2-8B, SharpDraft increases throughput over native SWA on Paragraph Ordering, while their throughput is similar on GPQA-D. This contrast is consistent with task-dependent reliance on distant context. On DSpark-8B, requests with larger matched-prefix acceptance gains from Full + SharpDraft over 2K SWA also tend to show larger full-generation MAL gains (r=0.91r=0.91; Figure 4).

Combining SWA and SharpDraft. The fixed-window comparisons show that SharpDraft can improve throughput when combined with SWA. Window size trades off access to distant context against attention cost, while scaling adjusts the retained keys and saturates once the visible-key count stops growing. Jointly tuning window size and reference cardinality for each workload remains future work.

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [2/2]

Weakness 2: Limited evidence on long-context-trained drafters. The primary DFlash models were trained with sequences of only 3,072 tokens but evaluated on much longer generations. It remains unclear whether SharpDraft primarily mitigates training–inference length mismatch or addresses a more general attention-dilution problem.

Question 2: Long-context training: Does SharpDraft remain effective when applied to drafters trained on longer sequences, particularly recent DFlash checkpoints trained with long-context data and sliding-window attention?

On the 40K-trained Qwen3.5-27B-DFlash, adding SharpDraft at a fixed 8K SWA window improves measured MAL and throughput on all four tasks (Section 4.4, Table 3). These comparisons include different continuations; their benefit should not be interpreted as an identical-output latency gain.

Training length and visible-key exposure are distinct: long sequences under SWA do not expose each query to the whole sequence. Comparing Query scaling at a fixed SWA window tests its effect under the same context-access constraint.

What the added evidence supports. The 40K-trained checkpoint results show that Query scaling can remain useful after long-context training. Separately, Appendix D.2 shows that uniform all-layer Query scaling at fixed states can reduce long-context draft–target mismatch.

This checkpoint comparison does not control for training length independently of model and attention configuration.


Weakness 3: Modest gains over fixed scaling. Table 3 shows that fixed query scaling already achieves most of the improvement, with dynamic scaling providing only an additional ~1% MAL gain.

Question 3: Dynamic vs. fixed scaling: Given the small improvement over the best fixed scale, could the authors further clarify when dynamic scaling is preferable in practice?

Dynamic scaling is useful when one configuration must handle varied requests without selecting a new fixed scale for each workload. Its additional mean throughput gains over each task’s fastest tested fixed scale are 0.53–2.15%. Across 120 requests, it reaches at least 98% of each request’s best measured fixed-setting throughput on 97.50% of requests, compared with at most 65.80% for any single fixed scale (Figure 6(c)).

Comparison across tasks. Table 4 reports all five fixed scales on AIME-26, GPQA-D, LongGenBench Block Short, and LongWeave Paragraph Ordering 4K. Each task uses 30 requests per method, including repetitive and capped outputs.

Full fixed-scale sweep (Table 4). Cells show throughput in tokens/s (MAL). Bold marks the best overall result; underlining marks the best fixed result (ties included).

MethodAIME-26GPQA-DBlock ShortParagraph 4K
Frozen (β=1\beta=1)91.33 (3.41)99.31 (3.25)165.59 (5.05)224.06 (6.84)
Fixed β=e0.1\beta=e^{0.1}113.34 (3.86)128.78 (3.57)229.67 (6.87)232.78‾\underline{232.78} (7.02‾\underline{7.02})
Fixed β=e0.2\beta=e^{0.2}129.69 (4.17)153.81 (3.82)264.46 (7.79)232.02 (6.93)
Fixed β=e0.3\beta=e^{0.3}136.83 (4.27)164.28 (3.92‾\underline{3.92})273.22‾\underline{273.22} (8.01‾\underline{8.01})226.26 (6.70)
Fixed β=e0.4\beta=e^{0.4}138.55‾\underline{138.55} (4.28‾\underline{4.28})165.92‾\underline{165.92} (3.92‾\underline{3.92})270.15 (7.96)216.86 (6.40)
Fixed β=e0.5\beta=e^{0.5}137.89 (4.23)164.43 (3.87)265.81 (7.78)206.91 (6.07)
SharpDraft139.28 (4.33)169.48 (4.02)277.32 (8.12)234.21 (7.03)

The preferred fixed scale can also change with generation length. On GPQA-D, the highest mean cumulative MAL is obtained with e0.1e^{0.1} at 4K, e0.3e^{0.3} at 16K, and e0.4e^{0.4} at 32K generated tokens. SharpDraft maintains top-32 attention mass near 0.76–0.77 over 2K–32K, while fixed-scale mass declines (Figure 6(a,b)). Each position averages only requests that reach it.

When to use dynamic scaling. The preferred constant changes across tasks, yet SharpDraft has the highest observed mean throughput on all four. The per-request comparison uses the best throughput among five measured fixed scales, selected retrospectively for each request. The observed coverage favors automatic adjustment for variable workloads; a stable workload can use a well-calibrated fixed scale.

The conclusion is limited to the evaluated greedy comparisons. The small mean timing differences do not establish statistical significance.

In summary, the same-window comparisons show that SharpDraft can complement SWA, including on the evaluated 40K-trained drafter. The task-dependent results clarify the benefit of access to distant context, while the fixed-scale analysis identifies varied workloads as a setting where dynamic scaling is useful.

Best regards,

Authors

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts and will maintain my Weak Accept score.

Dear reviewer CriU,

Thank you for your positive assessment and constructive feedback. We appreciate your time and thoughtful engagement throughout the discussion.

Best regards,

Authors

Reviewer_28z8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SharpDraft sharpens drafter attention by scaling post-normalization queries with the number of visible keys. The scale grows logarithmically above a reference count and is capped; weights, cached KVs and target verification remain unchanged. Applied to DFlash, PARD and EAGLE 3.1, it reports 2.59–3.19× geometric-mean end-to-end speedups over target-only decoding on three long-generation benchmarks.

Critical strengths
  • Minimal inference cost. The method needs no training, added parameters or optimizer state, and its measured peak allocation matches Frozen.
  • Controlled acceptance evidence. Identical outputs across all seven scaling settings on 30 AIME problems isolate a real gain in verification efficiency.
  • Transfer across drafters. Improvements extend beyond DFlash to both PARD and EAGLE 3.1.
Critical weaknesses
  • The motivation is not sufficiently validated. Declining top-k mass need not imply worse drafting. Why does dilution increase draft–target mismatch? Please compare target and drafter attention on matched contexts.
  • Practical relevance is unclear. Deployed DSpark [1] and released DFlash2 [2] drafters use sliding-window attention. What benefit remains with bounded key counts? Applicability to linear-attention drafters is also unaddressed.
  • Modest improvement. Fixed scaling achieves 4.023 MAL versus 4.063 (+0.99%) for dynamic scaling. End-to-end comparisons against fixed scaling and existing log-N scaling, including the cited scalable-softmax, are missing.

[1] Xin Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026. https://arxiv.org/abs/2607.05147.

[2] Z Lab. Qwen3.8-27B-DFlash2: Official Model Configuration, 2026. https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2/blob/main/config.json

Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Author Response [1/3]

Dear Reviewer 28z8,

Thank you for your careful review. We have added controlled attention interventions to test output mismatch directly and compared SharpDraft with SWA, fixed scaling, and SSMax. We have revised the manuscript and highlighted the changes in blue.


Weakness 1: The motivation is not sufficiently validated. Declining top-k mass need not imply worse drafting. Why does dilution increase draft–target mismatch? Please compare target and drafter attention on matched contexts.

We agree that declining top-k mass alone does not establish poorer drafting. On matched histories, the drafter shows a larger decline in attention concentration than the target, and scaling draft Queries reduces long-context output mismatch (Figure 5).

Matched-context comparison (Figure 5a; Appendix D.1). We compare Qwen3-8B and DFlash-b16 on shared greedy LGB Block Short histories. All 30 requests contain observed prefixes in all three length bands (2K, 8K, and 16K). We use 16 positions per request per band, totaling 1,440 states. For queries predicting the same next token, we measure top-32 attention mass, normalized within the shared historical prefix. Between 2K and 16K, this top-32 attention mass falls from 0.80 to 0.73 in the target and from 0.76 to 0.51 in the drafter. The drafter thus exhibits a larger decline in top-32 attention mass on matched histories, even though its total prefix attention mass increases from 0.74 to 0.80.

Controlled all-layer interventions (Figure 5b; Appendix D.2). Using SharpDraft’s scaling operator, we apply a common Query scale β\beta to all five draft layers while holding the decoding state and target distribution fixed. We measure full-vocabulary TV at temperature 1 and report ΔTV\Delta\mathrm{TV} relative to native β=1\beta=1; negative values indicate reduced mismatch.

At 16K, flattening with β=0.75\beta=0.75 yields ΔTV=+0.431\Delta\mathrm{TV}=+0.431 (95% CI: [0.407,0.454][0.407, 0.454]), whereas sharpening with β=1.25\beta=1.25 yields −0.271-0.271 [−0.294,−0.247][-0.294, -0.247]. For β=1.25\beta=1.25, ΔTV\Delta\mathrm{TV} is −0.074-0.074 [−0.089,−0.061][-0.089, -0.061] at 8K and +0.001+0.001 [−0.003,0.005][-0.003, 0.005] at 2K, with no clear 2K improvement. Intervals use paired request-level bootstrap resampling.

Interpretation. These interventions show that uniform all-layer Query scaling can reduce long-context draft–target output mismatch at fixed decoding states. The causal evidence concerns the scaling intervention: changes in the attention distribution within the prefix and in total prefix attention mass remain coupled. It therefore supports attention scaling as a contributor to reducing mismatch without establishing dilution as the sole cause of natural long-context degradation. These distributional results are distinct from live block MAL and end-to-end latency measurements.

The matched-context comparisons and interventions are now reported in Section 5.1 and Appendix D.

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [2/3]

Weakness 2: Practical relevance is unclear. Deployed DSpark [1] and released DFlash2 [2] drafters use sliding-window attention. What benefit remains with bounded key counts? Applicability to linear-attention drafters is also unaddressed.

A bounded SWA window reduces the opportunity for cardinality-dependent scaling, but does not necessarily eliminate it. At the same 8K window, adding SharpDraft to DFlash2-8B increases observed throughput by 0.37–1.27% across the four tasks below. We also compare unscaled Full, SWA, and Full + SharpDraft to distinguish changing context access from adding calibration (Section 4.4, Table 3).

RedHat DFlash2-8B uses native 2K SWA; DSpark-8B uses full attention by default, and we implement SWA on that released checkpoint. SWA removes direct access to older keys; Query scaling changes the weights of retained keys. We evaluate AIME-26, GPQA-D, LongGenBench Block Short, and LongWeave Paragraph Ordering at its official 4K tier, which requires reproducing and reordering source paragraphs.

DrafterMethodAIME-26GPQA-DBlock ShortParagraph 4K
DSpark-8BFull (Default)147.00 (4.25)170.00 (4.11)235.25 (6.62)220.48 (6.05)
DSpark-8BSWA (2K)150.87 (4.26)165.42 (3.97)163.76 (4.88)80.21 (2.26)
DSpark-8BSWA (8K)152.61 (4.32)172.25 (4.13)236.51 (6.65)225.41 (6.05)
DSpark-8BSWA (8K) + Ours153.33 (4.34)172.00 (4.13)236.83 (6.67)226.45 (6.07)
DSpark-8BFull + Ours152.34 (4.34)167.87 (4.14)230.58 (6.69)226.90 (6.07)
DFlash2-8BFull136.86 (3.97)170.11 (4.03)215.87 (6.17)126.71 (3.65)
DFlash2-8BSWA (2K, Default)140.69 (4.02)171.21 (4.02)176.63 (5.18)98.33 (2.76)
DFlash2-8BSWA (8K)140.18 (4.03)172.02 (4.06)221.43 (6.32)128.48 (3.65)
DFlash2-8BSWA (8K) + Ours141.84 (4.06)172.66 (4.07)224.12 (6.39)130.11 (3.70)
DFlash2-8BFull + Ours141.79 (4.06)173.04 (4.07)224.65 (6.39)130.45 (3.70)
DFlash-27B (40K-trained)Full50.41 (4.44)46.91 (5.07)50.63 (3.16)201.46 (12.96)
DFlash-27B (40K-trained)SWA (4K, Default)130.89 (7.85)100.09 (6.05)80.87 (4.95)183.95 (11.85)
DFlash-27B (40K-trained)SWA (8K)75.57 (5.04)75.28 (5.29)60.09 (3.77)195.59 (12.96)
DFlash-27B (40K-trained)SWA (8K) + Ours132.22 (7.98)97.95 (6.06)77.46 (4.96)205.80 (13.58)
DFlash-27B (40K-trained)Full + Ours131.11 (7.93)98.72 (6.03)79.40 (4.95)211.82 (13.58)

Table R4. Throughput in tokens/s (MAL). Full uses full draft attention without calibration; SharpDraft adds Query scaling. All repetitive and capped outputs are included. The 8B arms have matching outputs. SharpDraft uses N0=4KN_0=4\mathrm{K} for DSpark-8B and N0=3KN_0=3\mathrm{K} for DFlash2-8B and the 40K-trained DFlash. The 40K-trained DFlash comparisons include different continuations.

What the comparison shows. SWA remains competitive on the reasoning tasks. On DSpark, Full + SharpDraft achieves 2.83×2.83\times and 1.41×1.41\times the throughput of 2K SWA on Paragraph 4K and Block Short, respectively. Those comparisons change both context access and calibration. The unscaled Full rows and same-window SWA + SharpDraft rows separate these effects; the large gain over a small window should not be attributed entirely to Query scaling.

With bounded visible-key count, the scale saturates and remains one at or below N0N_0. The rule can therefore be applied within an SWA window, and Table 3 evaluates this combination with an 8K window. Purely linear-attention drafters are outside the softmax setting of our analysis and remain untested.

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [3/3]

Weakness 3: Modest improvement. Fixed scaling achieves 4.023 MAL versus 4.063 (+0.99%) for dynamic scaling. End-to-end comparisons against fixed scaling and existing log-N scaling, including the cited scalable-softmax, are missing.

Measured E2E comparisons show modest gains over the fastest fixed scales and closely comparable throughput to SSMax. SharpDraft achieves 0.53–2.15% higher throughput than each task’s fastest tested fixed scale; its throughput differs from SSMax by less than 1%. The tables below include both comparisons (Tables 4–5; protocols in Appendices C.1 and E).

Table 4 reports all five fixed scales across AIME-26, GPQA-D, LongGenBench Block Short, and LongWeave Paragraph Ordering 4K. Each task uses 30 requests per method, including repetitive and capped outputs.

Full fixed-scale sweep (Table 4). Cells show throughput in tokens/s (MAL). Bold marks the best overall result; underlining marks the best fixed result (ties included).

MethodAIME-26GPQA-DBlock ShortParagraph 4K
Frozen (β=1\beta=1)91.33 (3.41)99.31 (3.25)165.59 (5.05)224.06 (6.84)
Fixed β=e0.1\beta=e^{0.1}113.34 (3.86)128.78 (3.57)229.67 (6.87)232.78‾\underline{232.78} (7.02‾\underline{7.02})
Fixed β=e0.2\beta=e^{0.2}129.69 (4.17)153.81 (3.82)264.46 (7.79)232.02 (6.93)
Fixed β=e0.3\beta=e^{0.3}136.83 (4.27)164.28 (3.92‾\underline{3.92})273.22‾\underline{273.22} (8.01‾\underline{8.01})226.26 (6.70)
Fixed β=e0.4\beta=e^{0.4}138.55‾\underline{138.55} (4.28‾\underline{4.28})165.92‾\underline{165.92} (3.92‾\underline{3.92})270.15 (7.96)216.86 (6.40)
Fixed β=e0.5\beta=e^{0.5}137.89 (4.23)164.43 (3.87)265.81 (7.78)206.91 (6.07)
SharpDraft139.28 (4.33)169.48 (4.02)277.32 (8.12)234.21 (7.03)

The length-wise measurements in Figure 6(a,b) show that, on GPQA-D, e0.1e^{0.1}, e0.3e^{0.3}, and e0.4e^{0.4} give the highest mean cumulative MAL at 4K, 16K, and 32K generated tokens, respectively. Over 2K–32K, SharpDraft keeps top-32 attention mass near 0.76–0.77, whereas fixed-scale mass declines; each position averages only requests that reach it.

The preferred fixed scale changes across tasks, while SharpDraft uses one rule and achieves the highest observed throughput throughout. Across 120 requests, SharpDraft reaches at least 98% of each request’s best measured fixed-setting throughput on 97.50% of requests, versus at most 65.80% for any single fixed scale (Figure 6(c)). Its 0.53–2.15% additional throughput gains over the task-specific fastest fixed settings are modest; we do not claim statistical significance for these small differences.

End-to-end comparison with Scalable-Softmax (SSMax). We have added the requested comparison in Appendix E, Table 5. We use the same model, inputs, and decoding settings, with 30 requests per task.

The table reports output throughput (tokens/s), with MAL in parentheses.

MethodAIME-26GPQA-DLGB (Block Short)LongWeave (Paragraph 4K)
Frozen91.96 (3.41)97.35 (3.24)166.96 (5.05)225.10 (6.84)
SSMax139.97 (4.31)169.78 (3.99)279.59 (8.11)233.71 (6.96)
SSMax (β≥1\beta\geq1)140.21 (4.32)170.12 (4.00)279.57 (8.11)233.71 (6.96)
SharpDraft140.78 (4.33)171.44 (4.02)279.71 (8.12)235.42 (7.03)

Both SSMax and SharpDraft improve MAL and throughput over Frozen. Their throughput difference is less than 1% on all four tasks. These close results are consistent with both methods using log-N scaling to adjust draft attention as the context grows.

Behavior below the reference cardinality. In a separate evaluation across three drafters, the tested SSMax rule reduces mean MAL by 2.00–6.49% relative to Frozen for N<512N<512. SharpDraft retains β=1\beta=1 and matches Frozen throughout N≤N0N\leq N_0 (Appendix E.1, Figure 10).

In summary, the revised paper provides matched-context attention comparisons and controlled evidence of reduced draft–target mismatch, together with direct runtime comparisons against SWA, fixed scaling, and SSMax. These comparisons show task-dependent benefits over SWA, modest gains over the best fixed scales, and closely comparable throughput to SSMax.

Best regards,

Authors

Dear Reviewer 28z8,

Thank you again for your constructive review. As the discussion period is coming to a close, we would like to kindly follow up on whether our responses have addressed your concerns.

If any questions remain or further clarification would be helpful, please let us know. We would be happy to continue the discussion and improve our submission accordingly.

If your major concerns have been resolved, we would appreciate it if you could consider revising your score.

Best regards,

Authors

Reviewer_AQL6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SharpDraft, a training-free technique for improving speculative decoding under long contexts. The authors observe that attention becomes increasingly diluted as context length grows, potentially reducing draft acceptance. To address this, they introduce cardinality-aware query scaling, which adjusts attention sharpness based on the number of visible keys. The scaling rule is derived from a top-k attention-mass formulation and approximated using a fixed slope. Experiments with DFlash, PARD, and EAGLE 3.1 report improvements in mean accepted length (MAL) and end-to-end inference speed.

Critical strengths
  • The motivation is strong - declining draft acceptance during long-context generation is an important problem for efficient LLM inference.

  • The implementation overhead is low - the proposed query scaling requires neither retraining nor additional optimizer states, and the approach does not require modifying cached Keys and Values.

Critical weaknesses
  • Novelty is limited. Attention dilution and context-length-dependent temperature scaling have already been investigated, notably in Scalable-Softmax (https://arxiv.org/abs/2501.19399). SharpDraft mainly adapts this existing concept to speculative draft models. The top-k mass analysis provides a motivation for the scaling rule, but its assumptions do not establish a fundamentally new mechanism. I find the contribution incremental relative to existing attention-scaling techniques.

  • Although the method reports attractive speedups over autoregressive decoding, the more relevant gains over already optimized speculative decoding frameworks are considerably smaller, particularly for PARD and EAGLE 3.1 (in Table 2). The paper also lacks evaluations under realistic concurrent serving, batching, and stochastic decoding. The 100K-token experiments report acceptance rather than end-to-end latency. Consequently, the actual benefits for practical long-context inference remain unclear.

  • The central contribution provides only marginal improvement over simple fixed scaling. Table 3 reports an MAL of 4.023 for the best fixed scaling factor and 4.063 for SharpDraft, representing only ~1% improvement. Fixed scaling already captures almost all the benefit over the unmodified drafter. This substantially weakens the paper's central claim that cardinality-aware dynamic scaling is necessary or particularly effective. The practical significance of such a small gain remains questionable.

Questions for authors

See "Critical Weaknesses" for my concerns and questions.

5 comments
AuthorsOpenReview ↗

Author Response [1/3]

Dear Reviewer AQL6,

Thank you for your careful assessment and for recognizing the motivation and low overhead. We clarify the relationship to existing attention-scaling methods and provide direct comparisons that complement the calibration analysis and efficiency results. We have revised the manuscript and highlighted the changes in blue.


Weakness 1: Novelty is limited. Attention dilution and context-length-dependent temperature scaling have already been investigated, notably in Scalable-Softmax (https://arxiv.org/abs/2501.19399). SharpDraft mainly adapts this existing concept to speculative draft models. The top-k mass analysis provides a motivation for the scaling rule, but its assumptions do not establish a fundamentally new mechanism. I find the contribution incremental relative to existing attention-scaling techniques.

Our contribution is a formulation of attention dilution in speculative decoding and a training-free correction for long-context draft–target mismatch. Our scaling rule shares the logarithmic form of SSMax, which studies attention dilution and length-dependent scaling for length generalization. On matched histories, we find greater attention dilution in the drafter than in the target, and controlled Query-scaling interventions show that sharpening only the drafter’s attention reduces draft–target output mismatch at long contexts. The top-kk mass analysis motivates the scaling rule under its stated assumptions. End-to-end experiments show higher acceptance and throughput than the unscaled drafter, as well as higher throughput than the evaluated online adaptation baselines in our main evaluation.

1. Draft–target attention mismatch

We use the same 30 LGB Block Short requests for the 2K, 8K, and 16K comparisons. From 2K to 16K, top-32 attention mass, normalized within the shared historical prefix, falls by 0.25 in the drafter and 0.08 in the target. To investigate how attention dilution affects draft–target output mismatch in speculative decoding, we apply one common Query scale to all five draft layers and measure TV distance between draft and target next-token distributions, holding input prefixes and target distributions fixed.

At 16K, increasing the attention scale (β=1.25\beta=1.25) reduces TV by 0.27, whereas the same intervention provides no clear improvement at 2K (Figure 5). These experiments show that uniform all-layer Query scaling can reduce long-context draft–target mismatch; changes in within-prefix concentration and total prefix mass remain coupled.

2. Motivation for the scaling rule

As the reviewer recognizes, the top-kk mass analysis provides a motivation for the scaling rule. It separates the effect of the number of attended tokens (key count) from differences in attention scores (score contrast), explaining why attention scaling can counter dilution under the stated assumptions. This supports a computationally cheap form of context-length-dependent temperature scaling (cardinality-dependent scaling).

3. Acceptance and runtime

Tables 1–2 evaluate acceptance and E2E efficiency across DFlash, PARD, and EAGLE 3.1, including scaling costs. Additional E2E speedups over unmodified PARD and EAGLE 3.1 are 3.00–16.20% and 3.80–7.90%, respectively.

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [2/3]

Weakness 2: Although the method reports attractive speedups over autoregressive decoding, the more relevant gains over already optimized speculative decoding frameworks are considerably smaller, particularly for PARD and EAGLE 3.1 (in Table 2). The paper also lacks evaluations under realistic concurrent serving, batching, and stochastic decoding. The 100K-token experiments report acceptance rather than end-to-end latency. Consequently, the actual benefits for practical long-context inference remain unclear.

In the evaluated greedy settings, SharpDraft adds 3.00–16.20% E2E speedup to PARD and 3.80–7.90% to EAGLE 3.1, with calibration cost included. These gains are smaller than for DFlash. We report them relative to each unmodified speculative decoder, alongside measured 100K-session latency and concurrent-serving results below.

Gains over PARD and EAGLE 3.1. Table R1 restates the submitted Table 2 relative to each unmodified speculative decoder. Its final column reports the percentage speedup over each unmodified speculative decoder.

Draft frameworkBenchmarkUnmodified / AR+ SharpDraft / ARAdditional speedup
PARDAIME-262.69×2.69\times2.77×\mathbf{2.77}\times+3.00%
PARDGPQA-D3.39×3.39\times3.94×\mathbf{3.94}\times+16.20%
PARDLGB(Diary)2.73×2.73\times2.98×\mathbf{2.98}\times+9.00%
EAGLE 3.1AIME-261.92×1.92\times2.05×\mathbf{2.05}\times+6.40%
EAGLE 3.1GPQA-D2.53×2.53\times2.62×\mathbf{2.62}\times+3.80%
EAGLE 3.1LGB(Diary)3.52×3.52\times3.79×\mathbf{3.79}\times+7.90%

100K-token E2E latency. Table R2 reports end-to-end latency for the GLM-4.7-Flash sessions in Figure 3(d). Each of the 30 sessions per method generated 100,000 tokens across turns.

MethodMean session MALMean E2E time (s)Observed speedup over Frozen
Frozen drafter3.561,901.891.00×1.00\times
+ SharpDraft5.701,195.331.59×\mathbf{1.59}\times

The MAL and E2E gains closely track each other in these sessions. A 1.60×1.60\times increase in mean MAL accompanies a 1.59×1.59\times observed E2E speedup (37.15% lower session time). This agreement is consistent with the low cost of scalar computation and Query scaling: the acceptance gain is largely retained in measured runtime.

Table R2. Means over 30 sessions per method. Timing includes prefill, generation, calibration, and turn transitions, after model initialization. Both methods use the same question orders and 100K generated-token budget; continuations and turn boundaries may differ. All sessions reach the budget, and capped turns are included.

The full-session E2E speedups for all four models in Figure 3(a)–(d) are now reported in Section 4.3; mean session times are shown below.

ModelFrozen E2E time (s)SharpDraft E2E time (s)Speedup over Frozen
Qwen3-4B259.14150.321.72×1.72\times
Qwen3-8B306.51170.701.80×1.80\times
Llama-3.1-8B133.22111.651.19×1.19\times
GLM-4.7-Flash1901.891195.331.59×1.59\times

Batching and concurrent serving. We have added a greedy-decoding evaluation with vLLM at 1, 2, 4, and 8 concurrent requests, using 30 requests per task (Appendix F, Table 6). SharpDraft reduces mean request latency and increases aggregate throughput at every tested concurrency. Output tokens and termination match across methods and concurrency levels.

Table 6. Concurrent decoding. Each concurrency cell gives mean request E2E latency (s) / aggregate output throughput (tokens/s); CC is the number of concurrent requests.

TaskMethodMALC=1C=1C=2C=2C=4C=4C=8C=8
AIME-26Frozen3.41206.27 / 93.20214.04 / 175.60207.13 / 323.40187.18 / 573.70
AIME-26SharpDraft4.33133.12 / 144.30138.38 / 275.80131.50 / 540.10119.91 / 902.40
GPQA-DFrozen3.25107.48 / 99.50110.94 / 191.40112.17 / 342.2096.37 / 582.80
GPQA-DSharpDraft4.0261.68 / 173.4063.22 / 334.9063.57 / 626.6058.61 / 1095.30
LGB (Block Short)Frozen5.05136.62 / 170.80138.64 / 330.10132.28 / 677.80113.15 / 1365.90
LGB (Block Short)SharpDraft8.1281.33 / 286.9082.86 / 554.2078.15 / 1141.7067.48 / 2308.10
LongWeave (Paragraph 4K)Frozen6.8412.47 / 225.1012.59 / 438.0011.89 / 900.3010.74 / 1818.70
LongWeave (Paragraph 4K)SharpDraft7.0311.81 / 237.8011.91 / 466.4010.95 / 985.5010.32 / 1945.40

(Continued in the next comment)

AuthorsOpenReview ↗

Author Response [3/3]

Weakness 3: The central contribution provides only marginal improvement over simple fixed scaling. Table 3 reports an MAL of 4.023 for the best fixed scaling factor and 4.063 for SharpDraft, representing only ~1% improvement. Fixed scaling already captures almost all the benefit over the unmodified drafter. This substantially weakens the paper's central claim that cardinality-aware dynamic scaling is necessary or particularly effective. The practical significance of such a small gain remains questionable.

Dynamic scaling provides modest mean gains over a well-chosen fixed scale; its practical benefit is maintaining near-best throughput across varied requests. Across 120 requests, it reaches at least 98% of each request’s best measured fixed-setting throughput on 97.50% of requests, versus at most 65.80% for any single fixed scale (Figure 6(c)). The AIME-26 MAL gain over the best fixed scale is 1.22% (Table 4).

Comparison across tasks. Table 4 reports all five fixed scales on AIME-26, GPQA-D, LongGenBench Block Short, and LongWeave Paragraph Ordering 4K, using the same 30 requests per task for all methods. All repetitive and capped outputs are included.

Full fixed-scale sweep (Table 4). Cells show throughput in tokens/s (MAL). Bold marks the best overall result; underlining marks the best fixed result (ties included).

MethodAIME-26GPQA-DBlock ShortParagraph 4K
Frozen (β=1\beta=1)91.33 (3.41)99.31 (3.25)165.59 (5.05)224.06 (6.84)
Fixed β=e0.1\beta=e^{0.1}113.34 (3.86)128.78 (3.57)229.67 (6.87)232.78‾\underline{232.78} (7.02‾\underline{7.02})
Fixed β=e0.2\beta=e^{0.2}129.69 (4.17)153.81 (3.82)264.46 (7.79)232.02 (6.93)
Fixed β=e0.3\beta=e^{0.3}136.83 (4.27)164.28 (3.92‾\underline{3.92})273.22‾\underline{273.22} (8.01‾\underline{8.01})226.26 (6.70)
Fixed β=e0.4\beta=e^{0.4}138.55‾\underline{138.55} (4.28‾\underline{4.28})165.92‾\underline{165.92} (3.92‾\underline{3.92})270.15 (7.96)216.86 (6.40)
Fixed β=e0.5\beta=e^{0.5}137.89 (4.23)164.43 (3.87)265.81 (7.78)206.91 (6.07)
SharpDraft139.28 (4.33)169.48 (4.02)277.32 (8.12)234.21 (7.03)

Interpretation. The best fixed scale varies across workloads. SharpDraft uses the same cardinality-dependent rule across tasks and achieves the highest mean MAL and throughput on each task, with modest 0.53–2.15% additional throughput gains over each task’s fastest fixed setting.

On GPQA-D, the fixed scale with the highest mean cumulative MAL shifts from e0.1e^{0.1} at 4K to e0.3e^{0.3} at 16K and e0.4e^{0.4} at 32K generated tokens. SharpDraft maintains top-32 attention mass near 0.76–0.77 over 2K–32K, while fixed-scale attention mass declines (Figure 6(a,b)); each position averages only requests that reach it.

The request-wise reference is the best throughput among the five measured fixed scales for that request. It is selected retrospectively and does not represent a single fixed setting available without selection at deployment. Fixed scaling remains a competitive option for a stable workload; we do not claim statistical significance for the small timing differences.

In summary, matched-context comparisons and controlled interventions provide evidence that draft attention scaling can reduce long-context draft–target mismatch. SharpDraft improves measured runtime over unmodified speculative decoders in the evaluated greedy settings, including 100K-token sessions and concurrent decoding. Dynamic scaling maintains near-best throughput across requests with different preferred fixed scales.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.