acceptodds
Under review as a conference paper at ICLR 2027

TapKV: Accelerating Long-Horizon Speculative Decoding with Target-Aware Draft KV Caches Optimization

Abstract

The speedup of speculative decoding hinges on generating draft proposals at low cost while maintaining high draft–target agreement. In long-horizon speculative decoding, where generation spans long decoding sequences, however, the growing draft KV cache incurs substantial memory traffic and increases drafting latency. KV-cache pruning provides a natural opportunity to reduce this overhead and accelerate drafting. However, existing attention-based importance metrics primarily identify influential KV entries for draft model performance, without explicitly capturing how their removal affects draft–target agreement and, ultimately, the speculative acceptance rate. We introduce TapKV, a draft-cache pruning method. We derive an Attention-output Taylor (AOT) score that combines the deletion-induced attention-output change with the loss gradient, capturing both value removal and attention renormalization. AOT guides cache-entry selection and global budget allocation across layers, while a paged-attention kernel efficiently supports the resulting unequal cache lengths. Experiments across diverse workloads show that TapKV achieves higher acceptance lengths and speedups than existing baselines, delivering an average speedup of –.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Speculative Decoding, KV Cache Pruning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_d6i5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TapKV addresses the growing drafting cost in long-horizon speculative decoding by pruning the draft model’s KV cache while preserving accept length for draft model. Its main contribution is an Attention-output Taylor score that combines deletion-induced attention-output changes with gradients of a target guided loss to estimate KV cache importance. The method further introduces global cache-budget allocation across layers and a paged-attention backend that efficiently supports unequal per-layer cache lengths.

Critical strengths
  1. The authors seize upon the special objective of the draft model: cache pruning needs to preserve as much as possible the draft's ability to be accepted by the target. AOT combines the change in attention output caused by KV removal with the gradient of the target consistency loss, while considering both value removal and attention weight renormalization, forming an interpretable first-order sensitivity metric. At the system level: the authors combine cross-layer budget allocation, periodic scoring, and paged attention supporting different cache lengths per layer, addressing the execution overhead brought by adaptive cache allocation.
Critical weaknesses
  1. The experiments mainly compare vanilla SD, TriForce, and compare vanilla SD, TriForce, and SnapKV, StreamingLLM, and KeyDiff applied to the draft cache, lacking direct comparisons with the DFlash and EAGLE series.
Questions for authors
  1. Among the baselines compared by the authors, only TriForce is a speculative decoding method. StreamingLLM, SnapKV, and KeyDiff are all KV compression or KV pruning methods. Although the paper's method is not required to reach SOTA, the authors still need to compare with currently used speculative methods such as DFlash or DSpark on acceptance rate, latency, and throughput.

  2. The proposed method still uses a small model for speculative decoding.The overhead of such an autoregressive approach is non-trivial, and the accept length is very likely inferior to that of a single speculative head like DFlash. the training cost of a speculative head like DFlash is not high. The authors need to explain what advantages this work has over DFlash or multiple MTP, or what scenarios it is suitable for.

4 comments
AuthorsOpenReview ↗

Response to Reviewer d6i5

Thank you for the feedback on baseline selection and positioning. For quick reference, we summarize our main responses below.

  • Scope: TapKV prunes the draft KV cache and is designed for independent drafters. It does not change the drafter itself, so it is orthogonal to methods that improve the drafter, such as EAGLE, DFlash, and DSpark.
  • Baseline selection: No existing KV pruning method is designed for speculative decoding. We therefore compare with general KV eviction methods (SnapKV, StreamingLLM, and KeyDiff) applied to the draft cache, as well as TriForce.
  • TapKV on EAGLE-3, DFlash, and DSpark: TapKV can be applied on top of all three drafters. It improves the throughput and end-to-end speedup of each while keeping AL close to the unpruned drafter, for example from 1.37× to 1.54× on EAGLE-3 (chain) and from 2.88× to 3.50× on DSpark.

Response to Weakness 1

We agree that the EAGLE and DFlash series are widely used. However, they address a different problem from TapKV. EAGLE, DFlash, and DSpark are drafter designs that change how drafts are produced and trained. TapKV is a KV cache method that changes which draft KV entries are kept. Since no existing KV pruning method is designed for long horizon speculative decoding, we use general KV eviction methods applied to the draft cache as baselines.

To show how TapKV works with these drafters, we run EAGLE-3, DFlash, and DSpark in two configurations: the original drafter with the SDPA kernel (Vanilla), and the same drafter with TapKV on its paged kernel (+TapKV). The target model is Qwen3-8B (Table R1).

Table R1. TapKV on EAGLE-3, DFlash, and DSpark (target: Qwen3-8B).

DrafterModeMethodALTokens/s ↑SP ↑
EAGLE-3ChainVanilla2.29344.471.37×
+TapKV2.25850.241.54×
EAGLE-3TreeVanilla3.51958.451.80×
+TapKV3.46160.631.86×
DFlashChainVanilla2.83561.851.90×
+TapKV2.949107.743.31×
DSparkChainVanilla3.17893.752.88×
+TapKV3.104113.983.50×

TapKV improves the end-to-end speedup of every drafter while keeping AL close to the unpruned drafter. EAGLE-3 improves from 1.37× to 1.54× (chain) and from 1.80× to 1.86× (tree), DFlash from 1.90× to 3.31×, and DSpark from 2.88× to 3.50×.

Response to Question 1

DFlash, DSpark, and EAGLE-3 are draft architectures rather than KV cache methods, so they are complementary to TapKV, which can be applied on top of any of them. Table R1 reports acceptance length, throughput, and end-to-end speedup over target-only autoregressive decoding for all three, with and without TapKV. TapKV improves the throughput and speedup of every drafter while keeping AL close to the unpruned drafter, for example DSpark from 93.75 to 113.98 tokens/s (2.88× to 3.50×).

Response to Question 2

TapKV is orthogonal to the choice of drafter. It only requires a draft model with its own KV cache, and makes no assumption about the drafter's architecture or training. Table R1 shows that it applies directly to EAGLE-3, DFlash, and DSpark.

Independent drafters with TapKV are useful when training a target-specific draft head is impractical or not cost-effective. They cover cases that draft heads such as DFlash or MTP handle poorly:

  • New or updated targets. Even when training a draft head is cheap, the cost recurs: a head must be trained for each target and retrained every time the target is updated or fine-tuned. A small model from the same family (e.g., Qwen3-0.6B) works directly with any target that shares its tokenizer, with no training.
  • Long reasoning. An independent drafter keeps its own KV cache, which grows with the generated length and becomes its main cost in long reasoning. TapKV targets exactly this cost, and its benefit grows with longer generation and tighter budgets.
  • Memory- and bandwidth-limited devices. When on-chip memory or PCIe bandwidth is limited, the size of the draft KV cache decides whether and how efficiently the drafter can run. TapKV reduces this size directly.
AuthorsOpenReview ↗

Response Follow-up

Dear reviewer,

Thank you again for your detailed review!

As the discussion period is coming to an end, we would like to kindly check whether our response has addressed your concerns. We would be happy to answer any further questions.

The authors have addressed my questions and provided satisfactory responses. I indicated a rating of Weak Accept. However, the PDF does not appear to have been updated accordingly. I strongly encourage the authors to revise the PDF during the rebuttal period to incorporate these changes. That said, I tend to trust the authors will finalize the PDF properly. Best wishes.

AuthorsOpenReview ↗

Response to Final Decision

Thank you for your positive assessment and for updating your rating to Weak Accept. We appreciate the time you spent on our responses!

We are revising the paper to incorporate the changes discussed in the rebuttal. While doing so, we are also polishing the overall presentation, which has taken some additional time. We expect to upload the revised PDF later today (US Eastern Time).

Thank you again for your constructive feedback and support!

Reviewer_PfcQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TapKV reduces drafting costs during long-sequence speculative decoding by pruning the draft KV cache. It uses target feedback to select cache entries and allocate budgets across layers, supported by a paged-attention implementation. Experiments cover several model pairs, tasks, and decoding settings.

Critical strengths
  1. The paper addresses a practical bottleneck in long-sequence generation. Cache selection, layer-wise budgeting, and the implementation fit together well, while full target verification is retained.

  2. The experiments cover several models, tasks, and decoding settings. Tests on longer generation, different draft architectures, and pruning overhead provide useful evidence of practical benefits.

Critical weaknesses
  1. The contribution of the scoring method is difficult to separate from backend improvements. TapKV and the main baselines use different backends, which substantially affect speed. A same-backend comparison would help clarify the gains from scoring and budget allocation.

  2. The key ablations focus on one model–task setting and show relatively small gains. The separate benefits of target guidance and attention renormalization, and their consistency across samples, need stronger evidence. The scoring loss, replay procedure, and relationship between scores and actual pruning effects also need clarification.

Questions for authors
  1. With other settings fixed, how much do target guidance and attention renormalization each help, and are the gains consistent across samples?

  2. Under the same backend, cache budget, and pruning settings, how much advantage does TapKV retain over SnapKV and StreamingLLM?

  3. Could you clarify the scoring loss, replay procedure, and use of absolute values? How well do the scores reflect the impact of removing several cache entries together?

6 comments
AuthorsOpenReview ↗

Response to Reviewer PfcQ, Part 1

Thank you for the detailed questions on the backend comparison, the two mismatches, and the AOT score. For quick reference, we summarize our main responses below.

  • Same-backend comparison (W1, Q2): On the same paged attention kernel with identical settings, TapKV's AL is 7–28% higher than every baseline, and its speedup is 7–15% higher than the strongest baseline, in both settings (DeepSeek-R1-Distill-Qwen-1.5B/14B and Qwen3-0.6B/8B).
  • Robustness of Figure 4(b) (W2, Q1): Every ablated variant is worse than TapKV on 23 to 25 of 30 AIME problems (sign test p ≤ 0.005).
  • Objective mismatch (W2, Q1): Removing target guidance lowers AL at both budgets tested. The drop is larger at budget 256 (−1.57%), where the CI excludes zero.
  • Operation mismatch (W2, Q1): Without re-normalization, mean AL is 1.4% lower, and the variant is worse on 23 of 30 AIME problems (p = 0.005).
  • AOT loss definition (W2, Q3): The AOT loss is forward KL(target ‖ draft) over the full vocabulary plus the EAL loss on committed tokens, with W = 16 and λ = 1. AL changes by less than 1.5% for W from 8 to 32.
  • AOT score, loss, and AL (W2, Q3): When multiple tokens are removed together, the absolute-value AOT score correlates strongly with the actual loss change (ρ = 0.95), while the signed score does not (ρ = 0.30). A larger loss increase corresponds to a larger drop in actual AL.

Response to Weakness 1

Our contribution is not the paged-attention kernel itself, but how pruning is done on the paged cache. A vLLM-style block table is per request and shared by all layers, so a token must be kept or dropped in every layer at once. TapKV instead keeps a separate page table for each layer, so each layer evicts its own tokens under its own budget.

To separate this pruning design from the kernel, we run every method on the same paged-attention kernel. The baselines use the vLLM-style per-request page table, and TapKV uses its per-layer page table. All methods use the hyperparameters of our main experiments (Section 4, lines 349–354 and 361–363). The average draft-cache budget is Btot=1024B_{\mathrm{tot}} = 1024 tokens per layer and the recent buffer is R=1024R = 1024, so pruning is triggered whenever the draft cache reaches Bupper=Btot+R=2048B_{\mathrm{upper}} = B_{\mathrm{tot}} + R = 2048 (line 421, notation in Table 4, Appendix A.2). Attention-sink tokens, the recent window, and the replayed suffix are always kept (lines 238–241). We use chain drafting with γ = 7, greedy decoding, a maximum generation length of 8K tokens, and one RTX A6000. Speedup includes all scoring and pruning costs. For Table R1, TapKV's results are identical to Table 1 (DeepSeek-R1-Distill-Qwen-1.5B/14B, LongProc). Only the baselines were rerun, now on the same paged kernel instead of SDPA. For Table R2, we use Qwen3-0.6B (draft) and 8B (target) on LongProc with a 2K budget, and run all methods on the same paged kernel.

Table R1. Comparison on the same paged attention kernel (DeepSeek-R1-Distill-Qwen-1.5B/14B, LongProc).

MethodAL ↑SP ↑
TapKV5.322.57×
SnapKV4.452.13×
StreamingLLM4.792.24×
R-KV4.502.13×
KeyDiff4.161.96×

Table R2. Comparison on the same paged attention kernel (Qwen3-0.6B/8B, LongProc, budget 2K).

MethodAL ↑SP ↑
TapKV4.352.27×
SnapKV3.872.10×
StreamingLLM4.072.12×
R-KV3.792.07×
KeyDiff3.421.88×

In both tables, TapKV achieves 7–28% higher AL than every baseline, and its speedup is 7–15% higher than that of the strongest baseline.

AuthorsOpenReview ↗

Response to Reviewer PfcQ, Part 2

Response to Weakness 2

Loss definition. AOT uses the loss of Xia et al. (2026), which combines a distribution-matching term with an expected-acceptance-length (EAL) term. Let RW\mathcal{R}_W be the WW most recently committed positions, let yiy_i be the committed token at position ii, let PiP_i and QiQ_i be the full next-token distributions of the target and the draft, and let qi=Qi(yi)q_i = Q_i(y_i):

L=1W∑i∈RWKL(Pi∥Qi)+λLEAL\mathcal{L} = \frac{1}{W}\sum_{i\in\mathcal{R}_W}\mathrm{KL}(P_i \Vert Q_i) + \lambda \mathcal{L}_{\mathrm{EAL}},

LEAL=−∑k=1W∏j=1kqj\mathcal{L}_{\mathrm{EAL}} = -\sum_{k=1}^{W}\prod_{j=1}^{k} q_j.

The KL term is forward (target ‖ draft), taken over the full vocabulary and averaged over the WW positions. LEAL\mathcal{L}_{\mathrm{EAL}} uses only the draft probabilities of the committed tokens, and equals the negative expected number of consecutively accepted draft tokens. For the heterogeneous-vocabulary pair (Qwen3-0.6B → Phi-4-reasoning-14B), we use EAL as the loss function because they use different vocabulary. A committed token is one accepted by the target and appended to the output (lines 365–366). We use W=16W = 16 and λ=1\lambda = 1 in all experiments, and we will clarify this in Section 3.1 of the revised paper. Table R2 shows that AL changes by less than 1.5% for WW from 8 to 32. This study uses Qwen3-0.6B/8B on AIME, so its AL differs from Table R1.

Table R2. Replay window W (Qwen3-0.6B/8B, AIME).

WALTokens/s
84.99369.8
16 (default)5.04670.4
324.97469.4

The two mismatches. Both mismatches are defined in Section 2 (lines 142–146), and AOT addresses each one with a separate component (Section 3.2).

  • Objective mismatch → target-aware loss. Existing scores measure how much a token matters to the draft model itself, but speculative decoding needs the draft to agree with the target. AOT therefore takes gradients of a target-aware loss on recently verified tokens (Eqs. 2–3, lines 194–198 and 225–236).
  • Operation mismatch → deletion-aware output change. Attention-score heuristics ignore that deleting a token removes its value vector and renormalizes the remaining attention weights. AOT uses the exact output change caused by deletion (Eq. 1, lines 216–224).

Objective mismatch. With cache, budget, and backend fixed, we remove the target gradient from the AOT score, so tokens are scored only by the change in the draft's own output (Table R3).

Table R3. Effect of removing target guidance (Qwen3-0.6B/8B, AIME, 30 problems).

BudgetTapKVw/o targetΔAL [95% CI]Problems worse
10245.0464.977−1.36% [−2.69, +0.11]23/30
2564.3514.283−1.57% [−2.70, −0.43]20/30

AL is lower without target guidance at both budgets, and the drop is larger at budget 256, where the CI excludes zero.

Operation mismatch: re-normalization. When a token is deleted, the remaining attention weights are renormalized, which is the (1−a) term in Eq. (1). To isolate this step, we add a variant that estimates the output change as −a·v. It zeroes the deleted token's attention weight but does not renormalize the rest. In the Figure 4(b) setting (Qwen3-0.6B/8B, AIME 2025, budget 1024), this variant lowers mean AL from 5.046 to 4.978 (−1.4%) and is worse than TapKV on 23 of the 30 problems (paired sign test, p = 0.005, Table R4).

Robustness of Figure 4(b). Table R4 reports the per-problem AL distribution for each variant on AIME at budget 1024, with a paired two-sided sign test.

Table R4. Per-problem AL distribution (AIME, b1024, 30 problems).

VariantMean [95% CI]Medianp25 / p75Worse than TapKVSign test p
TapKV5.046 [4.861, 5.232]5.0154.642 / 5.408--
Gradient magnitude only4.942 [4.766, 5.117]4.8654.553 / 5.30025/30<0.001
w/o target4.977 [4.790, 5.162]4.9124.486 / 5.40623/300.005
w/o re-normalization (−a·v)4.978 [4.800, 5.158]5.0134.584 / 5.37023/300.005

Compared with TapKV, the mean AL is 2.1% lower with gradient magnitude only, 1.4% lower without target guidance, and 1.4% lower without re-normalization. Each variant is also worse than TapKV on 23–25 of the 30 problems, and all paired sign tests are significant (p ≤ 0.005).

AuthorsOpenReview ↗

Response to Reviewer PfcQ, Part 3

Response to Weakness 2 (continue)

AOT score, actual loss, and AL. We check this relationship in two steps.

First, we test whether the AOT score predicts the actual loss change when multiple tokens are removed together. We start from 8 AIME prompts with target-generated reasoning traces, prefill both models, and prune at several keep ratios using Σ|d| (TapKV), Σd (signed), and random selection. For each pruned cache, we record the change in each score and in the actual loss (Figure R1). Σ|d| correlates strongly with the actual loss change (Spearman ρ = 0.95), while Σd is nearly uncorrelated (ρ = 0.30). Since pruning evicts the lowest-scoring tokens, the low end matters most. The sets that Σ|d| predicts to be cheapest leave the loss nearly unchanged, whereas those Σd predicts to be cheapest still raise it by about 0.9. With signed sums, large positive and negative effects can cancel, so a harmful set of tokens can look harmless. Taking absolute values prevents this.

Second, we test whether the loss change reflects the actual AL change. In the same setting, we run 128 tokens of speculative decoding after pruning, using TapKV and random pruning at different pruning ratios (Figure R2). Figure R2 groups the pruned caches by their loss change (x-axis, percentile) and plots their AL relative to the full cache (y-axis), so 1.0 means pruning costs no AL. AL stays at about 1.0 for the smallest loss changes and falls to about 0.80 for the largest. A larger loss increase corresponds to a larger drop in actual AL (Spearman ρ = −0.72).

Reviewer_yKASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at a practical issue in speculative decoding: as generation gets longer, the draft model’s growing KV cache makes drafting increasingly expensive. TapKV uses a target-guided Taylor score to decide which KV entries to keep and how to divide the cache budget across layers. It also introduces a paged-attention implementation to support different cache lengths across layers. The experiments cover several model pairs and tasks, including greedy, sampling-based, and tree-based decoding. With tree drafting, the paper reports average speedups of 2.83–2.92× over target autoregressive decoding.

Critical strengths
  1. The idea makes sense for the problem. Drafting can become a bottleneck over long generations, so keeping the draft cache under control is useful. I also like the idea of using target feedback to choose what to retain, since agreement with the target is what ultimately matters for acceptance.

  2. There is a working system behind the idea. The authors implement support for different layer budgets and test the method with both chain and tree drafting, as well as stochastic sampling. The appendix also checks longer generation and pruning overhead. This gives some confidence that the approach can provide useful speedups in practice.

Critical weaknesses
  1. I am not yet sure how much of the speedup comes from the pruning method itself. Appendix B.2 says TapKV uses a paged kernel, whereas SnapKV and StreamingLLM use dense SDPA. Figure 4(c) shows that changing the backend alone, with a uniform budget, raises speedup from 1.70× to 2.05×. Adaptive allocation then brings it to 2.09×. I would therefore like to see the methods compared using the same backend. Given the focus on draft KV-cache pruning for long contexts, I would also expect comparisons with KV eviction methods designed for this setting, such as R-KV and TriAttention. Trying these methods on the draft side would help show how much target guidance adds.

  2. A few details at the core of AOT are still unclear to me. Section 3.1 defines p_i and q_i as probabilities of individual committed tokens, then mentions using KL divergence. I could not tell whether the actual loss uses full vocabulary distributions or only those token probabilities. I also could not find the replay-window size W used in the experiments. These details affect both the scores and their computation cost. Given that AOT improves speedup from 2.06× to 2.09× over the w/o-target variant in Figure 4(b), I would want these details settled before drawing a firm conclusion about the benefit of target guidance.

Questions for authors
  1. If SnapKV and StreamingLLM also use the same paged-attention backend, how much advantage does TapKV retain? It would help to see both acceptance length and end-to-end speedup, with pruning overhead included.

  2. Could you compare against R-KV and TriAttention applied to the draft cache, at least in one representative setting with matched cache budgets and the same backend? Given the focus on long generation, I think this comparison would help assess the value of target guidance. Please also explain any changes needed to adapt these methods.

  3. What exact loss is used to compute AOT, and what replay-window size W is used in the experiments? In particular, does the loss use full vocabulary distributions or only the probabilities of committed tokens? If it is KL divergence, please clarify its direction and how it is averaged.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yKAS, Part 1

Thank you for the careful reading of our backend setup, baselines, and AOT definition. For quick reference, we summarize our main responses below.

  • Same-kernel comparison (W1, Q1): With all methods running on the same paged attention kernel, TapKV achieves 7–28% higher AL than every baseline and a 7–13% higher speedup than the strongest baseline in both settings (DeepSeek-R1-Distill-Qwen-1.5B/14B and Qwen3-0.6B/8B). All speedups include scoring and eviction overhead under the same hyperparameters.
  • R-KV and TriAttention (Q2): We add both methods on the draft cache in two settings (Tables R1 and R4). TapKV achieves higher AL and speedup than both baselines.
  • AOT definition (W2, Q3): We give the exact objective, which is forward KL(target ‖ draft) over the full vocabulary plus the EAL loss on committed tokens. We use W = 16 and λ = 1. A sensitivity study shows that AL changes by less than 1.5% for W from 8 to 32.
  • Benefit of target guidance (W2): Removing the target term lowers AL on 23 of 30 AIME problems (sign test p = 0.005). At budget 256, the 95% CI excludes zero.

Response to Weakness 1

Our contribution covers both the scoring rule and how pruning is done on the paged cache. On top of a vLLM-style block table, which is per request and shared by all layers, a token must be kept or dropped in every layer at once. Since AOT scores differ across layers, TapKV keeps a separate page table for each layer, so each layer evicts its own tokens under its own budget.

To separate our contribution from the system setup, we run every method on the same paged kernel. The baselines use a single per-request page table, whereas TapKV employs the per-layer page-table design proposed in this paper. Unlike Table 1 of the paper, where the baselines use SDPA, all methods here share the paged kernel. The experiment uses DeepSeek-R1-Distill-Qwen-1.5B (draft) and 14B (target) (Table R1), and Qwen3-0.6B (draft) and 8B (target) (Table R2) on LongProc, and speedup is over target-only autoregressive decoding.

Table R1. Comparison on the same paged attention kernel (DeepSeek-R1-Distill-Qwen-1.5B/14B, LongProc).

MethodAL ↑SP ↑
KeyDiff4.161.96×
SnapKV4.452.13×
R-KV4.502.13×
TriAttention4.762.27×
StreamingLLM4.792.24×
TapKV5.322.57×

Table R2. Comparison on the same paged attention kernel (Qwen3-0.6B/8B, LongProc, budget 2K).

MethodAL ↑SP ↑
KeyDiff3.421.88×
R-KV3.792.07×
SnapKV3.872.10×
StreamingLLM4.072.12×
TapKV4.352.27×

In both tables, TapKV achieves 7–28% higher AL than every baseline and runs 7–13% faster than the strongest baseline.

Response to Weakness 2

Thank you for pointing out this ambiguity. We will write out the objective explicitly in Section 3.1 of the revised paper. Let pip_i and qiq_i be the next-token distributions of the target and draft models at position ii, and let yiy_i be the committed token at position ii. AOT scores tokens with the following loss over the WW most recent positions RW\mathcal{R}_W:

L=1W∑i∈RWKL(pi∥qi)+λLEAL\mathcal{L} = \frac{1}{W}\sum_{i\in\mathcal{R}_W} \mathrm{KL}(p_i \Vert q_i) + \lambda \mathcal{L}_{\mathrm{EAL}},

LEAL=−∑k=1W∏j=1kqj(yj)\mathcal{L}_{\mathrm{EAL}} = -\sum_{k=1}^{W}\prod_{j=1}^{k} q_j(y_j).

The first term is the forward KL from the target to the draft, computed over the full vocabulary and averaged over positions. The second term is the expected accepted length (EAL) loss of Xia et al. (2026) [1], which uses only the draft probability of each committed token. For the heterogeneous-vocabulary pair (Qwen3-0.6B → Phi-4-reasoning-14B), we use EAL as the loss function because they use different vocabulary. We use W=16W = 16 and λ=1\lambda = 1 in all experiments. Table R3 shows that AL changes by less than 1.5% for WW from 8 to 32. This study uses Qwen3-0.6B/8B on AIME, so its AL differs from Tables R1 and R2.

Table R3. Replay window W (Qwen3-0.6B/8B, AIME).

WALTokens/s
84.99369.8
16 (default)5.04670.4
324.97469.4

On computation cost, each pruning event replays the W = 16 most recent tokens once to compute the scores. In our Qwen3-0.6B/8B experiments, one scoring pass takes about 75–80 ms, and 8K-token generation triggers 230 to 910 pruning events in total across the 30 problems of a dataset. This cost is included in all reported speedups.

On the benefit of target guidance, removing the target term lowers AL on 23 of 30 AIME problems at budget 1024 (sign test p = 0.005). At budget 256, AL drops by 1.57% [−2.70, −0.43].


[1] Xia, Tianhua, et al. "Acceptance-Aware Draft Model Training for Speculative Decoding." arXiv, 21 Sept. 2026, arXiv:2609.24150, https://doi.org/10.48550/arXiv.2609.24150.

AuthorsOpenReview ↗

Response to Reviewer yKAS, Part 2

Response to Question 1

On the same paged kernel with DeepSeek-R1-Distill-Qwen-1.5B/14B and a 1K budget, TapKV reaches AL 5.32 and a 2.57× speedup, compared with AL 4.45 / 2.13× for SnapKV and AL 4.79 / 2.24× for StreamingLLM (Table R1). On Qwen3-0.6B/8B with a 2K budget, TapKV reaches AL 4.35 and a 2.27× speedup, compared with AL 3.87 / 2.10× for SnapKV and AL 4.07 / 2.12× for StreamingLLM (Table R2). All speedups include scoring and eviction overhead. Our response to Weakness 1 gives the full setup. All methods share the same paged kernel, budget, and hyperparameters, and differ only in the scoring rule and page table: the baselines use a per-request page table, and TapKV uses per-layer page tables.

Response to Question 2

We adapt R-KV and TriAttention to the draft cache using their official scoring rules, under the same budget and page table as the other baselines. On the same paged kernel, TapKV achieves 18% and 12% higher AL than R-KV and TriAttention, and a 2.57× speedup compared with 2.13× and 2.27× (Table R1). We also compare R-KV and TriAttention in the setting of our main experiments (Qwen3-0.6B/8B, averaged over AIME, GPQA, LiveCodeBench, and LongProc, budget 1024). TapKV reaches AL 3.58 and a 1.99× speedup, compared with AL 3.42 / 1.61× for R-KV and AL 3.54 / 1.64× for TriAttention (Table R4).

Table R4. R-KV and TriAttention on the draft cache (Qwen3-0.6B/8B, average over AIME, GPQA, LiveCodeBench, and LongProc, budget 1024; baselines use SDPA).

MethodAL ↑SP ↑
R-KV3.421.61×
TriAttention3.541.64×
TapKV3.581.99×

Response to Question 3

AOT uses forward KL(target ‖ draft) over the full vocabulary plus the EAL loss on the draft probabilities of committed tokens, averaged over the W = 16 most recent positions with λ = 1. AL remains stable for W from 8 to 32 (Table R3). The full formula is in our response to Weakness 2.

AuthorsOpenReview ↗

Response Follow-up

Dear reviewer,

Thank you again for your detailed review!

As the discussion period is coming to an end, we would like to kindly check whether our response has addressed your concerns. We would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.