acceptodds
Under review as a conference paper at ICLR 2027

DLite: History-Free Speculative Decoding for Agentic AI

18 views

Abstract

Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to gener- ate candidate tokens that are subsequently verified by the target model. How- ever, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the his- torical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra- block causal dependencies with low overhead. To effectively train this heteroge- neous history-free architecture, we introduce a tailored two-stage training strat- egy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2∼6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, Speculative Decoding, Model Design and Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZqihOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DLite removes the drafter’s historical KV cache by conditioning on target hidden states from a single sequence position across multiple layers. It combines a parallel backbone, a lightweight recurrent head, and two-stage distillation, and evaluates decoding efficiency on general, long-context, and agentic workloads.

Critical strengths
  1. Removing the drafter’s historical KV cache offers a clear opportunity to reduce cache maintenance overhead.

  2. The paper reports both acceptance length and decoding speed, with ablations on training and architectural choices.

Critical weaknesses
  1. The central experiments do not adequately distinguish length generalization from architectural superiority. DLite is trained with a maximum sequence length of 4k, while some evaluations use 40k–118k input tokens. The baselines’ training lengths, checkpoint provenance, and long-context adaptation are insufficiently documented.

  2. If all methods were trained on short sequences (which appears likely, although the paper does not clearly specify this), these results would primarily establish robustness to length extrapolation under that training regime. They would not substantiate the broader explanation that maintaining historical context causes baseline degradation. Comparisons against long-context-trained or adapted baselines are missing.

  3. The KV-cache ablation covers only GSM8K, MBPP, and Alpaca. It therefore does not establish whether removing history remains beneficial on the long-context and agentic workloads central to the paper.

Questions for authors
  1. Which checkpoints are used for each baseline, and what are their training datasets, maximum sequence lengths, and training budgets?

  2. Why are long-context-trained or adapted baselines absent from an evaluation targeting long-context workloads? Does DLite retain its advantage under matched adaptation data and budgets?

  3. Can the authors compare No KV Cache, SWA-128, and SWA-512 on long-context and agentic workloads, with results broken down by input length?

7 comments
AuthorsOpenReview ↗

Thank you for your constructive feedback.

Q1

Thank you for requesting these details. We use publicly released checkpoints for EAGLE-3, DFlash, DSpark, and DFlash2; DFlow is sourced from its original paper. The sources and reported training configurations are summarized below.

MethodSourceReported training configuration
EAGLE-3AngelSlimFollowing the original paper: ShareGPT (approximately 68K examples) and UltraChat-200K (approximately 464K examples), with responses regenerated by the target model. Reference
DFlashz-labApproximately 800K examples from NVIDIA Nemotron Post-Training Dataset V2 and CodeAlpaca, with target-generated responses; 6 training epochs; maximum training sequence length of 3072 tokens for the Qwen3-4B/8B models. Reference
DSparkDeepSeekOpen-PerfectBlend (1.3M examples), with responses regenerated by each target model in non-thinking mode; 10 training epochs. Reference
DFlash2RedHatAIQwen3-8B-Regenerated-Collection, combining instruction-following, chat, math, and code datasets; 1 training epoch; training sequence length of 8192 tokens, as documented in the checkpoint’s model card.
DFlowOriginal paper80K ShareGPT examples; 6 training epochs.

Q2

Our evaluation focuses on generalization from short-sequence training to long-context workloads. We use the released baseline checkpoints without additional long-context adaptation and evaluate all methods under consistent experimental settings. Under this setting, DLite achieves substantial improvements on long-context workloads, supporting the effectiveness of its architectural design for length generalization. We expect DLite to further benefit from training on longer generated responses. However, such training incurs substantially higher computational costs. In the future, we will extend training to longer sequences and release the corresponding checkpoints.

Q3

We have extended the history ablation to long-context and agentic workloads on Qwen3-4B, comparing SWA-128, SWA-512, and DLite. Each entry reports the average acceptance length τ\tau, followed by the speedup over autoregressive decoding.

WorkloadTeacherSWA-512SWA-128DLite
Multi-DocQA(32k~64k)2.13(1.71×)2.80(2.20×)2.75(2.09×)2.74(2.18×)
LongIn-ctxLrn.(32k~64k)2.45(2.05×)3.23(2.54×)3.12(2.32×)3.11(2.58×)
CodeRepo.Ud.(32k~64k)1.88(1.50×)3.25(2.68×)3.19(2.54×)3.14(2.66×)
Long-dialogHist.(32k~64k)3.04(2.62×)4.73(3.89×)4.65(3.78×)4.62(3.88×)
AgentMemBench(80k~120k)1.72(1.28×)2.85(2.40×)2.80(2.28×)2.80(2.39×)
AgentLongBench(80k~120k)1.31(1.09×)2.58(2.08×)2.53(2.00×)2.51(2.07×)
BFCL-v3(32k~64k)1.84(1.45×)2.88(2.41×)2.79(2.29×)2.78(2.37×)
SWE-Bench(80k~120k)1.73(1.47×)3.27(2.71×)3.18(2.62×)3.15(2.68×)

The two SWA variants and DLite achieve comparable acceptance lengths and decoding performance, supporting bounded history as an effective design principle. Our choice of none history additionally accounts for memory efficiency and infrastructure simplicity. In prefix-caching serving setup, the SWA variants restrict the attention window but still retain draft KV states for the complete prefix to support reuse across requests and agent turns. Thus, a bounded attention window does not translate into an equally bounded historical-cache footprint. DLite eliminates this historical draft KV storage altogether, removing its contribution to peak GPU memory and providing more memory headroom for concurrent requests.

DLite also eliminates the corresponding draft-cache updates after verification and the retention and management of historical draft states across turns. This simplifies integration with prefix caching and multi-turn serving, reducing the infrastructure work needed to maintain an additional history-dependent component. Given the similar acceptance lengths, we choose the history-free design to combine competitive drafting quality and decoding speed with lower memory requirements and simpler serving infrastructure.

Reviewer_ZqihOpenReview ↗

Maintain my original rating

Thank you for the response. Q2 confirms that the paper’s claims exceed what the experiments establish. The results demonstrate performance under short-sequence training, but do not establish an inherent disadvantage of history-aware drafting. Long-context adaptation has already shown benefits for existing methods. However, evaluating such adapted baselines is a natural comparison for a paper targeting long-context workloads. Omitting this comparison leaves the central architectural claim unsupported. I will maintain my original rating.

AuthorsOpenReview ↗

Thank you for your constructive feedback.

Thank you for emphasizing this comparison. We agree that evaluating adapted baselines is important and have now conducted additional experiments. We fine-tuned both DSpark and DLite on the same 4,000 training examples, each approximately 20K tokens long, using identical fine-tuning settings.

The following table reports average acceptance length on LongBench-v2 with 32–64K input tokens. “FT” denotes long-context fine-tuning.

BenchmarkDSparkDSpark FTDLiteDLite FT
Multi-DocQA2.192.642.742.86
LongIn-ctxLrn.2.412.883.113.18
CodeRepo.Ud.2.442.903.143.20
Long-dialogHist.3.934.324.624.75

Long-context adaptation improves DSpark across all four tasks, consistent with the reviewer’s observation. DLite also benefits from adaptation and retains higher acceptance lengths across all four tasks under the same fine-tuning settings.

We further evaluated both adapted models on examples with 100–120K input tokens:

BenchmarkDSpark FTDLite FTRelative improvement in acceptance length
LongBench-v22.282.7219.3%
SWE-Bench2.132.8332.9%

These results address the missing comparison: DLite’s advantage persists after long-context adaptation of the history-aware baseline, including when extrapolating substantially beyond the adaptation length. This supports the effectiveness of DLite’s architecture for length generalization, alongside its draft KV-cache memory savings and simpler infrastructure integration.

Reviewer_xC8fOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing speculative drafters lose much of their acceptance length on long, multi-turn agentic contexts and must maintain a draft side KV cache. DLite removes draft-side history entirely. A bidirectional backbone consumes three inputs, multi layer target hidden states, the anchor token, and K mask placeholders, all at local positions. A low-rank RNN head then restores intra-block causality in hidden space. Training has two stages, KL distillation from a history-aware isomorphic teacher, then CE+TV alignment with the target. DLite matches or slightly exceeds DSpark on general benchmarks and substantially outperforms it on long context and agentic workloads.

Critical strengths
  • Large, consistent gains exactly where current drafters fail. Acceptance-length collapse on long agentic contexts is a practically important problem. DLite’s improvements over DSpark hold across two model sizes, four LongBench-v2 subsets, and four agentic benchmarks, without sacrificing quality on general workloads.
  • A simple and well motivated design. Delegating global context to the target model and making drafting a local, length invariant problem is a clean idea that also removes draft KV cache maintenance. The design space study (number of target layers, composition strategy, head rank) supports the final configuration.
Critical weaknesses

The long context gains may partly reflect how the baselines were trained. If DSpark and the other drafters were trained only on short sequences, their degradation at 40k–118k tokens could stem from never having seen long inputs, rather than from an inherent inability of history dependent drafters to model long context. The paper does not report the baselines’ training data or maximum sequence length. The history ablation (Figure 5b) is evaluated only on short benchmarks, so this alternative explanation is not yet ruled out.

Questions for authors
  • How were the baseline drafters trained (data, maximum sequence length, budget)? If they saw only short sequences, can you report DSpark trained with long-context inputs, or the SWA-128/512 variants and the history-aware teacher, on SWE-Bench, AgentLongBench, and LongBench-v2?
  • Please explain the role of the learnable depth embedding in more detail. Since the multi layer hidden states appear to share local position 0, is its main purpose to identify the source layer, or to distinguish hidden states from token embeddings?
  • Which positions’ target hidden states does the drafter consume each round, the anchor only, or all tokens accepted in the previous round?
  • Citations in Sections 1,2 are run into the text without parentheses and should use proper formatting. MBPP is also cited to Austin et al. (2021), which is the D3PM diffusion paper.
3 comments
AuthorsOpenReview ↗

Thank you for your constructive feedback.

Thank you for your constructive feedback.

Q1

All baseline draft models evaluated in our paper use officially released checkpoints available on Hugging Face. According to their technical reports, these models are trained on short sequences, with generated responses typically limited to 4096 tokens. The comparison is therefore fair in terms of response-generation length. The training data sources are also similar, with most models using NVIDIA’s Nemotron Post-Training Dataset V2 and the publicly available Open-PerfectBlend dataset. To ensure a fair comparison, we restricted our training to these datasets and did not perform additional training specifically targeting long inputs or long outputs.

Moreover, I can provide the training results of Teacher Model, SWA-512, and SWA-128 on long context and agent Benchmarks on Qwen3-4B.

WorkloadTeacherSWA-512SWA-128DLite
Multi-DocQA2.13(1.71×)2.80(2.20×)2.75(2.09×)2.74(2.18×)
LongIn-ctxLrn.2.45(2.05×)3.23(2.54×)3.12(2.32×)3.11(2.58×)
CodeRepo.Ud.1.88(1.50×)3.25(2.68×)3.19(2.54×)3.14(2.66×)
Long-dialogHist.3.04(2.62×)4.73(3.89×)4.65(3.78×)4.62(3.88×)
AgentMemBench1.72(1.28×)2.85(2.40×)2.80(2.28×)2.80(2.39×)
AgentLongBench1.31(1.09×)2.58(2.08×)2.53(2.00×)2.51(2.07×)
BFCL-v31.84(1.45×)2.88(2.41×)2.79(2.29×)2.78(2.37×)
SWE-Bench1.73(1.47×)3.27(2.71×)3.18(2.62×)3.15(2.68×)

The two SWA variants and DLite achieve comparable acceptance lengths and decoding performance, supporting bounded history as an effective design principle. Our choice of none history additionally accounts for memory efficiency and infrastructure simplicity. In prefix-caching serving setup, the SWA variants restrict the attention window but still retain draft KV states for the complete prefix to support reuse across requests and agent turns. Thus, a bounded attention window does not translate into an equally bounded historical-cache footprint. DLite eliminates this historical draft KV storage altogether, removing its contribution to peak GPU memory and providing more memory headroom for concurrent requests.

DLite also eliminates the corresponding draft-cache updates after verification and the retention and management of historical draft states across turns. This simplifies integration with prefix caching and multi-turn serving, reducing the infrastructure work needed to maintain an additional history-dependent component. Given the similar acceptance lengths, we choose the history-free design to combine competitive drafting quality and decoding speed with lower memory requirements and simpler serving infrastructure.

Q2

The learnable depth embedding serves both purposes: it identifies the source layer of each target hidden state and distinguishes these target features from ordinary token embeddings.

Specifically, for a hidden state ht(l)h_t^{(l)} extracted from target layer ll, we construct zt(l)=ht(l)+edepth(l)z_t^{(l)}=h_t^{(l)}+e_{\mathrm{depth}}^{(l)} where edepth(l)e_{\mathrm{depth}}^{(l)} is a learnable, layer-specific embedding.

The target hidden states share local position 0 because they represent the same sequence position across different layers. Their positional encoding therefore does not distinguish their source layers. Depth embeddings provide this complementary layer identity, allowing the backbone to distinguish and combine features from different depths. Adding them to the target features also differentiates these inputs from the anchor and draft-placeholder token embeddings.

We will clarify these two roles and their distinction from local positional encoding in Section 4.1.1.

Q3

At each speculative round, DLite consumes target hidden states from a single sequence position across multiple selected layers, rather than from all tokens accepted in the previous round.

Specifically, we use the last position in the committed prefix that has been processed by the target model. In the first round, this is the final prompt token; in subsequent rounds, it is the corresponding position selected after verification. The new anchor token is supplied separately through its token embedding. If the anchor is denoted by xtx_t, the contextual features are ht−1l,l∈Lh_{t-1}^l, l\in\mathcal{L}.

Thus, the multiple hidden-state inputs represent different layers at the same position, not different historical positions. Earlier accepted tokens contribute through the context already encoded in these target features, without being retained as additional historical inputs to the drafter. We will clarify this position alignment in Section 4.1 and Figure 3.

Q4

Thank you for pointing out these errors. We will correct the citation formatting in Sections 1 and 2 to use appropriate parenthetical citations. We will also replace the incorrect MBPP reference to the D3PM paper with Austin et al. (2021), “Program Synthesis with Large Language Models.”

Once again, I would like to thank you for your very constructive suggestions.

AuthorsOpenReview ↗

Thank you for your constructive feedback.

We further extend the additional experiments. We fine-tuned both DSpark and DLite on the same 4,000 training examples, each approximately 20K tokens long, using identical fine-tuning settings.

The following table reports average acceptance length on LongBench-v2 with 32–64K input tokens. “FT” denotes long-context fine-tuning.

BenchmarkDSparkDSpark FTDLiteDLite FT
Multi-DocQA2.192.642.742.86
LongIn-ctxLrn.2.412.883.113.18
CodeRepo.Ud.2.442.903.143.20
Long-dialogHist.3.934.324.624.75

Long-context adaptation improves DSpark across all four tasks, consistent with the reviewer’s observation. DLite also benefits from adaptation and retains higher acceptance lengths across all four tasks under the same fine-tuning settings.

We further evaluated both adapted models on examples with 100–120K input tokens:

BenchmarkDSpark FTDLite FTRelative improvement in acceptance length
LongBench-v22.282.7219.3%
SWE-Bench2.132.8332.9%

These results address the missing comparison: DLite’s advantage persists after long-context adaptation of the history-aware baseline, including when extrapolating substantially beyond the adaptation length. This supports the effectiveness of DLite’s architecture for length generalization, alongside its draft KV-cache memory savings and simpler infrastructure integration.

I think additional experiments shows the soundness of the paper enough. I will maintain my rating as weak accept.

Reviewer_JiuvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DLite addresses a weakness of speculative decoding: draft models become less accurate and more costly as context grows, especially in agentic tasks. Instead of maintaining its own history and KV cache, DLite uses the target model’s hidden states to predict a block of tokens, with a small RNN head capturing dependencies within the block. It matches strong baselines on short-context tasks and performs better on long-context and agentic workloads. On SWE-Bench with Qwen3-8B, it raises speedup from DSpark’s 1.60× to 2.74×.

Critical strengths

The problem is relevant and timely as the lengths of context in LLM keep growing.

DLite gives large benefits on long context length and does not need to maintain KV cache for the draft model.

Critical weaknesses

It seems there is a mismatch between the central claim that drafter’s history is not necessary and the evidence. Sections 1 and 3.2 argue that a growing history makes prediction harder for a smaller drafter, but in the evidence it mixes several factors. Figure 1(b) compares different tasks, so input length and task type change together. DLite itself drops from 6.38 on GSM8K to 3.15 on SWE-Bench (Qwen3-4B), which suggests that task type accounts for much of the drop. The baselines' training lengths are not reported, while the test inputs (32K to 118K tokens) are 8 to 29 times longer than DLite's 4096-token training sequences, so part of the baselines' degradation may come from this train-test length mismatch. There is also a missing step between diagnosis and solution. The diagnosis only says the history should not keep growing, and a fixed window satisfies that too. The SWA-128 and SWA-512 variants in Section 5.3 could test this, but they are only evaluated on GSM8K, MBPP and Alpaca, where contexts are short, so they say nothing about the long-input regime that motivates the paper. This is the main factor in my assessment, because it decides whether the contribution is a new principle (keep no draft history) or a narrower point (keep the draft window bounded). Also, Medusa (Cai et al., 2024) and Hydra (Ankner et al., 2024) already draft from the target's features at the current position without a draft-side history, and the paper should explain how DLite differs from them. Also the agent and system-efficiency claims are not measured directly.

Questions for authors
  1. Extending the ablation in Section 5.3, what are the acceptance length and speedup of the four draft models (the history-aware teacher from Section 4.2, SWA-512, SWA-128 and DLite) on the long-context and agentic benchmarks in Tables 2 and 3? Section 5.3 only tests GSM8K, MBPP and Alpaca, which have short contexts, so it cannot tell whether DLite's long-context gains come from removing the draft history or only from bounding it. If SWA-512 matches DLite on long inputs, the design principle would be "bounded history" rather than "no history". If both the teacher and the SWA variants degrade while DLite does not, that would address my main concern. Since these models are already trained, this should be cheap to run.

  2. Currently the experiments are not conducted end to end on benchmarks. How about the performance on real multi-round benchmarks with prefix cache on?

  3. According to Eq. 4, at pos T, DLite needs the actual word at pos T-1. It seems that LM heads are needed for every step. Does DLite use the full vocabulary for every LMhead? Please report the latency for each step of the draft model. Also the acceptance lengths for the same model and benchmark differ between Table 1 and Table 4. For LiveCodeBench on Qwen3-4B, DLite's τ is 5.31 in Table 1 but 5.98 in Table 4, while DSpark only moves from 5.20 to 5.28. Were the two tables measured under different settings, and which numbers should be used to judge the throughput gains in Table 4?

2 comments
AuthorsOpenReview ↗

Thank you for your constructive feedback.

Thank you for your constructive feedback.

Q1

We agree that the original short-context ablation did not sufficiently distinguish bounded history from no history. We have extended the evaluation to the history-aware teacher, SWA-512, SWA-128, and DLite across the long-context and agentic workloads in Tables 2 and 3 on Qwen3-4B. Each entry reports average acceptance length τ\tau, followed by decoding speedup over autoregressive decoding in parentheses.

WorkloadTeacherSWA-512SWA-128DLite
Multi-DocQA2.13(1.71×)2.80(2.20×)2.75(2.09×)2.74(2.18×)
LongIn-ctxLrn.2.45(2.05×)3.23(2.54×)3.12(2.32×)3.11(2.58×)
CodeRepo.Ud.1.88(1.50×)3.25(2.68×)3.19(2.54×)3.14(2.66×)
Long-dialogHist.3.04(2.62×)4.73(3.89×)4.65(3.78×)4.62(3.88×)
AgentMemBench1.72(1.28×)2.85(2.40×)2.80(2.28×)2.80(2.39×)
AgentLongBench1.31(1.09×)2.58(2.08×)2.53(2.00×)2.51(2.07×)
BFCL-v31.84(1.45×)2.88(2.41×)2.79(2.29×)2.78(2.37×)
SWE-Bench1.73(1.47×)3.27(2.71×)3.18(2.62×)3.15(2.68×)

SWA-512, SWA-128, and DLite achieve comparable acceptance lengths across all eight workloads, substantially exceeding those of the teacher. Their decoding performance is also close: DLite achieves speeds comparable to SWA-512 and consistently higher than SWA-128. We agree that these findings support restricting draft-side history.

Our choice of none history additionally accounts for memory efficiency and infrastructure simplicity. In prefix-caching serving setup, the SWA variants restrict the attention window but still retain draft KV states for the complete prefix to support reuse across requests and agent turns. Thus, a bounded attention window does not translate into an equally bounded historical-cache footprint. DLite eliminates this historical draft KV storage altogether, removing its contribution to peak GPU memory and providing more memory headroom for concurrent requests.

DLite also eliminates the corresponding draft-cache updates after verification and the retention and management of historical draft states across turns. This simplifies integration with prefix caching and multi-turn serving, reducing the infrastructure work needed to maintain an additional history-dependent component. Given the similar acceptance lengths, we choose the history-free design to combine competitive drafting quality and decoding speed with lower memory requirements and simpler serving infrastructure.

Q2

For fixed inputs and decoding settings, prefix caching reuses existing KV states without changing the conditioning context or mathematically equivalent model predictions. The expected acceptance length is therefore unchanged. Prefix caching reduces redundant prefill computation and TTFT, while speculative decoding reduces target-model forward passes during decoding. In our evaluated workloads, prefill contributes only a small fraction of inference latency, so prefix caching is expected to have a limited effect on overall speedup while preserving DLite’s decoding benefits. Our BFCL evaluation already executes 41 multi-turn sessions. The reported measurements aggregate model calls exceeding 32K input tokens and exclude tool-execution time.

Q3

DLite uses the full vocabulary at every draft-token prediction step. Eq. (4) consumes the preceding draft token, with the anchor providing the initial conditioning. DLite performs sequential refinement in hidden space before the final vocabulary projection, eliminating the separate intermediate base LM-head projection.

Component latencies are shown below in μs. DSpark’s Markov-head measurements include its LM-head matrix multiplication.

BatchSizeBaseLMHead(DFlash)MarkovHead(DSpark)SequentialHead(DLite)Backbone
14199196662195
84339847322261
3249711678272668
6489216259153192
1281924288911395450

DLite’s head is faster than DSpark’s at every tested batch size. At batch size 1, it adds only 0.247 ms over DFlash’s direct LM head; at batch size 128, it is faster than both alternatives.

Tables 1–3 use Hugging Face, whereas Table 4 uses SGLang. Different low-level kernels introduce floating-point numerical differences that can affect token selection and acceptance decisions. Table 4’s throughput must therefore be interpreted using Table 4’s acceptance lengths: 5.98 for DLite and 5.28 for DSpark on Qwen3-4B/LiveCodeBench. We will identify the frameworks explicitly in the captions.

Finally, we will clarify the relationship to Medusa and Hydra, which already use target features and, in Hydra’s case, sequentially dependent predictions. DLite’s distinction is its combination of a parallel backbone over multiple target-layer features at local positions, a low-rank recurrent head, and two-stage distillation, evaluated on long-context and agentic workloads.

Once again, I would like to thank you for your very constructive suggestions.

AuthorsOpenReview ↗

Thank you for your constructive feedback.

We further extend the additional experiments. We fine-tuned both DSpark and DLite on the same 4,000 training examples, each approximately 20K tokens long, using identical fine-tuning settings.

The following table reports average acceptance length on LongBench-v2 with 32–64K input tokens. “FT” denotes long-context fine-tuning.

BenchmarkDSparkDSpark FTDLiteDLite FT
Multi-DocQA2.192.642.742.86
LongIn-ctxLrn.2.412.883.113.18
CodeRepo.Ud.2.442.903.143.20
Long-dialogHist.3.934.324.624.75

Long-context adaptation improves DSpark across all four tasks, consistent with the reviewer’s observation. DLite also benefits from adaptation and retains higher acceptance lengths across all four tasks under the same fine-tuning settings.

We further evaluated both adapted models on examples with 100–120K input tokens:

BenchmarkDSpark FTDLite FTRelative improvement in acceptance length
LongBench-v22.282.7219.3%
SWE-Bench2.132.8332.9%

These results address the missing comparison: DLite’s advantage persists after long-context adaptation of the history-aware baseline, including when extrapolating substantially beyond the adaptation length. This supports the effectiveness of DLite’s architecture for length generalization, alongside its draft KV-cache memory savings and simpler infrastructure integration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.