- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DLite removes the drafter’s historical KV cache by conditioning on target hidden states from a single sequence position across multiple layers. It combines a parallel backbone, a lightweight recurrent head, and two-stage distillation, and evaluates decoding efficiency on general, long-context, and agentic workloads.
Critical strengths
-
Removing the drafter’s historical KV cache offers a clear opportunity to reduce cache maintenance overhead.
-
The paper reports both acceptance length and decoding speed, with ablations on training and architectural choices.
Critical weaknesses
-
The central experiments do not adequately distinguish length generalization from architectural superiority. DLite is trained with a maximum sequence length of 4k, while some evaluations use 40k–118k input tokens. The baselines’ training lengths, checkpoint provenance, and long-context adaptation are insufficiently documented.
-
If all methods were trained on short sequences (which appears likely, although the paper does not clearly specify this), these results would primarily establish robustness to length extrapolation under that training regime. They would not substantiate the broader explanation that maintaining historical context causes baseline degradation. Comparisons against long-context-trained or adapted baselines are missing.
-
The KV-cache ablation covers only GSM8K, MBPP, and Alpaca. It therefore does not establish whether removing history remains beneficial on the long-context and agentic workloads central to the paper.
Questions for authors
-
Which checkpoints are used for each baseline, and what are their training datasets, maximum sequence lengths, and training budgets?
-
Why are long-context-trained or adapted baselines absent from an evaluation targeting long-context workloads? Does DLite retain its advantage under matched adaptation data and budgets?
-
Can the authors compare No KV Cache, SWA-128, and SWA-512 on long-context and agentic workloads, with results broken down by input length?
7 comments
Maintain my original rating
Thank you for the response. Q2 confirms that the paper’s claims exceed what the experiments establish. The results demonstrate performance under short-sequence training, but do not establish an inherent disadvantage of history-aware drafting. Long-context adaptation has already shown benefits for existing methods. However, evaluating such adapted baselines is a natural comparison for a paper targeting long-context workloads. Omitting this comparison leaves the central architectural claim unsupported. I will maintain my original rating.
Thank you for your constructive feedback.
Thank you for emphasizing this comparison. We agree that evaluating adapted baselines is important and have now conducted additional experiments. We fine-tuned both DSpark and DLite on the same 4,000 training examples, each approximately 20K tokens long, using identical fine-tuning settings.
The following table reports average acceptance length on LongBench-v2 with 32–64K input tokens. “FT” denotes long-context fine-tuning.
| Benchmark | DSpark | DSpark FT | DLite | DLite FT |
|---|---|---|---|---|
| Multi-DocQA | 2.19 | 2.64 | 2.74 | 2.86 |
| LongIn-ctxLrn. | 2.41 | 2.88 | 3.11 | 3.18 |
| CodeRepo.Ud. | 2.44 | 2.90 | 3.14 | 3.20 |
| Long-dialogHist. | 3.93 | 4.32 | 4.62 | 4.75 |
Long-context adaptation improves DSpark across all four tasks, consistent with the reviewer’s observation. DLite also benefits from adaptation and retains higher acceptance lengths across all four tasks under the same fine-tuning settings.
We further evaluated both adapted models on examples with 100–120K input tokens:
| Benchmark | DSpark FT | DLite FT | Relative improvement in acceptance length |
|---|---|---|---|
| LongBench-v2 | 2.28 | 2.72 | 19.3% |
| SWE-Bench | 2.13 | 2.83 | 32.9% |
These results address the missing comparison: DLite’s advantage persists after long-context adaptation of the history-aware baseline, including when extrapolating substantially beyond the adaptation length. This supports the effectiveness of DLite’s architecture for length generalization, alongside its draft KV-cache memory savings and simpler infrastructure integration.
Thank you for your constructive feedback.
Q1
Thank you for requesting these details. We use publicly released checkpoints for EAGLE-3, DFlash, DSpark, and DFlash2; DFlow is sourced from its original paper. The sources and reported training configurations are summarized below.
Q2
Our evaluation focuses on generalization from short-sequence training to long-context workloads. We use the released baseline checkpoints without additional long-context adaptation and evaluate all methods under consistent experimental settings. Under this setting, DLite achieves substantial improvements on long-context workloads, supporting the effectiveness of its architectural design for length generalization. We expect DLite to further benefit from training on longer generated responses. However, such training incurs substantially higher computational costs. In the future, we will extend training to longer sequences and release the corresponding checkpoints.
Q3
We have extended the history ablation to long-context and agentic workloads on Qwen3-4B, comparing SWA-128, SWA-512, and DLite. Each entry reports the average acceptance length τ, followed by the speedup over autoregressive decoding.
The two SWA variants and DLite achieve comparable acceptance lengths and decoding performance, supporting bounded history as an effective design principle. Our choice of none history additionally accounts for memory efficiency and infrastructure simplicity. In prefix-caching serving setup, the SWA variants restrict the attention window but still retain draft KV states for the complete prefix to support reuse across requests and agent turns. Thus, a bounded attention window does not translate into an equally bounded historical-cache footprint. DLite eliminates this historical draft KV storage altogether, removing its contribution to peak GPU memory and providing more memory headroom for concurrent requests.
DLite also eliminates the corresponding draft-cache updates after verification and the retention and management of historical draft states across turns. This simplifies integration with prefix caching and multi-turn serving, reducing the infrastructure work needed to maintain an additional history-dependent component. Given the similar acceptance lengths, we choose the history-free design to combine competitive drafting quality and decoding speed with lower memory requirements and simpler serving infrastructure.