- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TapKV addresses the growing drafting cost in long-horizon speculative decoding by pruning the draft model’s KV cache while preserving accept length for draft model. Its main contribution is an Attention-output Taylor score that combines deletion-induced attention-output changes with gradients of a target guided loss to estimate KV cache importance. The method further introduces global cache-budget allocation across layers and a paged-attention backend that efficiently supports unequal per-layer cache lengths.
Critical strengths
- The authors seize upon the special objective of the draft model: cache pruning needs to preserve as much as possible the draft's ability to be accepted by the target. AOT combines the change in attention output caused by KV removal with the gradient of the target consistency loss, while considering both value removal and attention weight renormalization, forming an interpretable first-order sensitivity metric. At the system level: the authors combine cross-layer budget allocation, periodic scoring, and paged attention supporting different cache lengths per layer, addressing the execution overhead brought by adaptive cache allocation.
Critical weaknesses
- The experiments mainly compare vanilla SD, TriForce, and compare vanilla SD, TriForce, and SnapKV, StreamingLLM, and KeyDiff applied to the draft cache, lacking direct comparisons with the DFlash and EAGLE series.
Questions for authors
-
Among the baselines compared by the authors, only TriForce is a speculative decoding method. StreamingLLM, SnapKV, and KeyDiff are all KV compression or KV pruning methods. Although the paper's method is not required to reach SOTA, the authors still need to compare with currently used speculative methods such as DFlash or DSpark on acceptance rate, latency, and throughput.
-
The proposed method still uses a small model for speculative decoding.The overhead of such an autoregressive approach is non-trivial, and the accept length is very likely inferior to that of a single speculative head like DFlash. the training cost of a speculative head like DFlash is not high. The authors need to explain what advantages this work has over DFlash or multiple MTP, or what scenarios it is suitable for.
4 comments
Response Follow-up
Dear reviewer,
Thank you again for your detailed review!
As the discussion period is coming to an end, we would like to kindly check whether our response has addressed your concerns. We would be happy to answer any further questions.
The authors have addressed my questions and provided satisfactory responses. I indicated a rating of Weak Accept. However, the PDF does not appear to have been updated accordingly. I strongly encourage the authors to revise the PDF during the rebuttal period to incorporate these changes. That said, I tend to trust the authors will finalize the PDF properly. Best wishes.
Response to Final Decision
Thank you for your positive assessment and for updating your rating to Weak Accept. We appreciate the time you spent on our responses!
We are revising the paper to incorporate the changes discussed in the rebuttal. While doing so, we are also polishing the overall presentation, which has taken some additional time. We expect to upload the revised PDF later today (US Eastern Time).
Thank you again for your constructive feedback and support!
Response to Reviewer d6i5
Thank you for the feedback on baseline selection and positioning. For quick reference, we summarize our main responses below.
Response to Weakness 1
We agree that the EAGLE and DFlash series are widely used. However, they address a different problem from TapKV. EAGLE, DFlash, and DSpark are drafter designs that change how drafts are produced and trained. TapKV is a KV cache method that changes which draft KV entries are kept. Since no existing KV pruning method is designed for long horizon speculative decoding, we use general KV eviction methods applied to the draft cache as baselines.
To show how TapKV works with these drafters, we run EAGLE-3, DFlash, and DSpark in two configurations: the original drafter with the SDPA kernel (Vanilla), and the same drafter with TapKV on its paged kernel (+TapKV). The target model is Qwen3-8B (Table R1).
Table R1. TapKV on EAGLE-3, DFlash, and DSpark (target: Qwen3-8B).
TapKV improves the end-to-end speedup of every drafter while keeping AL close to the unpruned drafter. EAGLE-3 improves from 1.37× to 1.54× (chain) and from 1.80× to 1.86× (tree), DFlash from 1.90× to 3.31×, and DSpark from 2.88× to 3.50×.
Response to Question 1
DFlash, DSpark, and EAGLE-3 are draft architectures rather than KV cache methods, so they are complementary to TapKV, which can be applied on top of any of them. Table R1 reports acceptance length, throughput, and end-to-end speedup over target-only autoregressive decoding for all three, with and without TapKV. TapKV improves the throughput and speedup of every drafter while keeping AL close to the unpruned drafter, for example DSpark from 93.75 to 113.98 tokens/s (2.88× to 3.50×).
Response to Question 2
TapKV is orthogonal to the choice of drafter. It only requires a draft model with its own KV cache, and makes no assumption about the drafter's architecture or training. Table R1 shows that it applies directly to EAGLE-3, DFlash, and DSpark.
Independent drafters with TapKV are useful when training a target-specific draft head is impractical or not cost-effective. They cover cases that draft heads such as DFlash or MTP handle poorly: