acceptodds
Under review as a conference paper at ICLR 2027

SparseDFlash: Dynamic Context Retrieval for Long-video Speculative Decoding

Abstract

Block-diffusion drafters accelerate speculative decoding through parallel token prediction, but repeatedly attending to long video contexts can make drafting a bottleneck to overall speedup. Existing approaches reduce this overhead by re- stricting the draft context through sliding-window attention or visual token prun- ing. However, permanently discarding context can remove evidence needed for subsequent predictions. This limitation is particularly relevant to long videos, where successive predictions may depend on different, temporally distant events. We introduce SparseDFlash, an attention framework for efficient long-video spec- ulative decoding that reuses the drafter’s existing attention parameters without in- troducing additional learnable parameters. The framework preserves the complete context while dynamically restricting attention to evidence relevant to each draft- ing step. This allows the drafter to revisit distant evidence as generation progresses without repeatedly attending to the entire context. We further address a limitation of naive raster-order block construction, which can separate spatiotemporally ad- jacent visual tokens across retrieval blocks. Locality-aware ordering based on a three-dimensional Hilbert curve organizes these blocks according to the video’s spatial and temporal structure. The framework preserves the target model’s output distribution through standard speculative verification. Experiments on long-video description tasks using videos from four video understanding benchmarks show that SparseDFlash achieves state-of-the-art performance across all four bench- marks with both Qwen3-VL-8B and Qwen3.5-9B, while consistently outperform- ing the full-attention drafter using the same checkpoint.

Primary area:
optimization
Keywords:
Speculative Decoding, Vision Language Models, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JStgOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SparseDFlash uses sparse attention to speed up long-video speculative decoding. It keeps the full context but selects only relevant blocks for each drafting step. The selection gives more weight to early draft positions. Visual tokens are reordered to group nearby regions across frames. The method adds no learnable parameters and needs no extra training. Experiments on four video datasets show faster generation than full-attention DFlash with the same drafter weights.

Critical strengths
  1. Dynamic retrieval reduces context access while keeping distant video evidence available.
  2. The method adds no learnable parameters and requires no extra training. On Qwen3-VL-8B, it runs 1.6–1.7× faster than DFlash on LongVideoBench.
Critical weaknesses
  1. The novelty over prior work needs clearer justification. STD [1] already uses sparse drafting and dense verification for video understanding. DyCoke [2] allows visual KV entries to be reactivated during decoding. Jenga [3] and DFSAttn [4] already use 3D Hilbert reordering for video sparse attention. The paper should clarify what its parallel drafting and prefix-aware routing add to these designs.
  2. The speedup on Qwen3.5-9B is small. At temperature 1, the average speedup over autoregressive decoding is only 1.049×. The authors attribute this to verification overhead in the GatedDeltaNet layers. However, no latency breakdown supports this explanation. More detailed measurements would help identify the main bottleneck.

[1] Xuan Zhang et al. Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs. ACL, 2025.

[2] Keda Tao et al. DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models. CVPR, 2025.

[3] Yuechen Zhang et al. Training-Free Efficient Video Generation via Dynamic Token Carving. NeurIPS, 2025.

[4] Jie Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation. arXiv:2605.23445

Questions for authors
  1. What is the main technical advance over STD, DyCoke, and prior Hilbert-based sparse attention methods?
  2. How much latency comes from drafting and verification on Qwen3.5-9B? Do these measurements support the explanation for its limited speedup?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JStg (1/2)

We thank the reviewer for the careful review and the pointers to related work. We add all four papers to Related Work.

W1 / Q1. Technical advance over STD, DyCoke and Hilbert-based sparse attention

Main advance. SparseDFlash adapts dynamic block retrieval to a frozen parallel drafter, with selection matched to prefix acceptance. A shared block set must support jointly predicted positions, but a later candidate can be accepted only if all earlier candidates are accepted. We therefore compute relevance separately for each position and aggregate it with exponentially decaying prefix weights (Section 4.2).

Within SparseDFlash, replacing these weights with uniform position weights lowers τ by 0.242 on VDC (Table 2). Tables 4–5 demonstrate applicability to official DFlash checkpoints without additional training.

  • STD (Zhang et al., ACL 2025). Both methods draft sparsely and verify densely. STD uses the target itself to draft autoregressively, requiring γ sequential full-target passes per cycle. Its visual KV entries are selected once at prefill using averaged text-query attention. SparseDFlash instead sparsifies a lightweight parallel drafter and re-selects blocks at every drafting step, keeping previously unselected evidence reachable.
  • DyCoke (Tao et al., CVPR 2025). DyCoke prunes and reactivates entries in the target's KV cache without full-context verification, so it does not guarantee preservation of the target distribution. SparseDFlash restricts only the drafter, and the target verifies every token using the full context. Our contrast with “permanently discarding context” concerns existing video speculative drafting methods, including SpecVLM, ViSpec and Sparrow.
  • Jenga (Zhang et al., NeurIPS 2025) and DFSAttn (Hu et al., arXiv:2605.23445). Both use Hilbert orderings for sparse attention in video-generation DiTs. Jenga also mean-pools key blocks and selects blocks per query block, so these are shared foundations. Its scoring uses the mean query, while SparseDFlash aggregates position-specific relevance distributions with prefix weights. The setting also differs: the DiTs attend bidirectionally over a fixed latent grid within a denoising step, whereas our context grows with generated text. Our visual ordering uses C-frame chunks fixed at prefill and an index map inside the fused kernel, without physically reordering the KV cache (Appendix A).
AuthorsOpenReview ↗

Response to Reviewer JStg (2/2)

W2 / Q2. Qwen3.5-9B latency breakdown

On Qwen3.5-9B, verifying a 16-token block costs about two AR steps. GatedDeltaNet accounts for almost all of the additional cost.

We profile vanilla DFlash on the same 50 VDC clips, using approximately 9K-token inputs, T=0 and up to 512 generated tokens under the Table 1 configuration. We synchronize around each phase and use CUDA events for timings by layer type. These instrumented measurements are reported separately from the end-to-end results.

All timings below are in milliseconds, and τ is tokens advanced per cycle. “Cycle” denotes DFlash, and “AR” denotes one autoregressive step.

Phase / metricQwen3-VL-8B: cycleQwen3-VL-8B: ARQwen3.5-9B: cycleQwen3.5-9B: AR
Drafting3.25.8
Target forward22.522.662.831.2
├ GatedDeltaNet––47.116.0
├ Attention15.213.44.64.6
├ MLP3.03.53.43.3
└ Embeddings, norms and LM head4.37.7
Cache / state restoration0.10.9
Other: sampling, acceptance and bookkeeping0.91.0
Total per cycle26.770.5
τ3.652.77

Attention and MLP remain close to their single-step costs on both targets. GatedDeltaNet, however, costs 2.94× a single step during verification. Our verifier updates the recurrent state token by token so it can restore any accepted prefix. Restoration itself is inexpensive. We observe the same pattern at T=1, where GatedDeltaNet verification costs 2.8× a single step.

This approximately 9K-token VDC profile identifies target verification as the dominant cost on the hybrid model. SparseDFlash reduces the draft-side context-access cost, with Table 1 showing improvements in end-to-end speedup of about 9–11% over DFlash on the longer benchmarks at T=0.

Verification-side optimizations. HybridSpec (Wang et al., arXiv:2611.04055, Nov. 2026) addresses the same bottleneck on hybrid GatedDeltaNet-MoE targets through adaptive verification lengths, parallel processing of the selected prefix and efficient state reconstruction. These verification-side optimizations are potentially complementary to SparseDFlash's reduction of drafting cost. We add the breakdown to Appendix E and discuss the verification bottleneck in the revision.

Thanks for taking a close look at how this work differs from prior methods. The new latency breakdown also clears up my second concern. It shows that most of the time is spent on target verification, with GatedDeltaNet accounting for nearly all of the extra forward-pass cost. This helps explain why speeding up the drafter gives only a small overall gain on Qwen3.5-9B. I’m satisfied with this explanation and have decided to raise my score from 2 to 3.

Reviewer_qiixOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SparseDFlash accelerates long-video speculative decoding by replacing full-context drafter attention with parameter-free, query-dependent block retrieval. It retains the complete KV cache, exposes an initial block and recent window, and retrieves remote blocks using prefix-weighted queries. A 3D Hilbert ordering improves spatiotemporal locality. Full-context target verification preserves exact outputs. Across four caption-style video workloads and two model families, SparseDFlash outperforms full-attention DFlash. Ablations examine retrieval budgets, routing, token ordering, pretrained drafters, and model scale.

Critical strengths

1 Training-Free, Tailored Algorithmic Design: The paper cleverly adapts sparse attention to the unique mechanics of block-diffusion speculative decoding. Prefix-aware routing aligns with prefix-only verification, while 3D Hilbert reordering preserves spatiotemporal locality during mean-pooling. Crucially, this is achieved without extra parameters or training, making it a plug-and-play upgrade for pre-trained drafters. This strong algorithmic fit and practicality heavily support acceptance. 2. Hardware-Aware Implementation and Strong Empirical Gains: Rather than just simulating sparsity, the authors implement a fused Triton kernel that avoids intermediate KV gathering. This translates theoretical FLOP reductions into real wall-clock speedups (up to 1.7x faster than full-attention DFlash on long videos). Furthermore, showing that sparse retrieval can occasionally increase acceptance length over full attention (by filtering out noise) is a compelling, consequential finding.

Critical weaknesses

The experiments replace original benchmark queries (e.g., Video-MME, LongVideoBench) with a single generic description prompt across only 50 videos per dataset. While long responses are needed to measure decoding speed, generic captioning follows a predictable chronological pattern. It does not test whether dynamic retrieval can accurately locate specific, distant needle-in-a-haystack evidence required by complex, targeted user queries.

Questions for authors

Since evaluation uses a single generic description prompt, how does SparseDFlash perform on targeted, query-focused long-video QA (e.g., multi-hop reasoning requiring long-form answers) where relevant visual evidence is sparse and non-chronological?

1 comment
AuthorsOpenReview ↗

Response to Reviewer qiix

We thank the reviewer for the positive assessment and for highlighting the need to evaluate targeted queries.

W1 / Q1. Query-focused long-video QA and multi-hop answers

We evaluate MINERVA (Nagrani et al., arXiv:2505.00681, 2025), whose questions require multiple reasoning skills over evidence distributed across a video. We use the Qwen3-VL-8B target, checkpoint and SparseDFlash configuration from Table 1, at T=0.

We sample videos with a fixed seed, with one question per video, and evaluate 50 questions. The videos range from 81 seconds to 1.7 hours, with a median duration of 6.1 minutes. The prompt asks for open-ended answers that decompose the question, cite timestamps and combine the findings, with a maximum of 4,096 tokens. The full prompt appears in Appendix D.2.

Outputs that collapse into repetition inflate acceptance, so we skip such questions and report the first 50 questions in sampling order with non-collapsed outputs.

Method (50 questions)Output length (tokens, median)τEnd-to-end speedup vs. AR
DFlash5173.0001.346×
SparseDFlash5173.0121.976×

SparseDFlash maintains DFlash's acceptance length with a higher end-to-end speedup. It reads 2,560 keys from a median context of approximately 44K tokens. These results support the effectiveness of sparse drafting on query-focused multi-hop workloads.

We also evaluate the original Video-MME questions with a reasoning prompt on the 50 Table 1 videos. SparseDFlash matches DFlash's accuracy with token-identical outputs and a higher end-to-end speedup. Both evaluations are added to Appendix D.2.

Retrieval beyond recency. Under the original description prompt, remote retrieval with a smaller window achieves higher acceptance than a larger recent window at all three context lengths, despite reading fewer keys (Table 6). This complements the query-driven evaluation.

Reviewer_uurXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SparseDFlash reduces repeated context access in block-diffusion speculative drafting. It retains the complete draft context but dynamically retrieves remote KV blocks at each drafting step, alongside an initial block and a recent window. Prefix-aware aggregation prioritizes early draft positions, and 3D Hilbert/Gilbert ordering constructs more local visual retrieval blocks. A fused kernel avoids an intermediate gathered KV buffer. Target verification remains unchanged. The proposed inference modifications introduce no additional trainable parameters, and comparisons with full-attention DFlash use identical drafter weights. Evaluation uses long video-description workloads and additional pretrained text and multimodal checkpoints.

Critical strengths
  1. The paper targets a clear bottleneck: repeated full-context reads by a parallel drafter. Keeping context available for later retrieval avoids irreversible evidence removal.

  2. The identical-checkpoint DFlash comparison controls for training differences and provides direct evidence that sparse context access improves the measured latency–acceptance trade-off. The work includes a fused kernel, measured speedups, context-budget ablations, and tests on official pretrained checkpoints. The degradation from replacing all sliding-window layers is also reported, providing a useful applicability boundary.

Critical weaknesses
  1. The independent benefits of prefix-aware aggregation and Hilbert ordering beyond generic dynamic block retrieval remain incompletely established. Current component ablations are primarily on VDC and show relatively small acceptance-length gains. Stronger same-budget retrieval comparisons and component-level wall-clock results on long video contexts would strengthen the contribution.

  2. The main workloads use a common video-description prompt rather than the original QA tasks. Generalization to realistic extended-reasoning outputs remains unclear. For suitable targets, report final-answer accuracy, end-to-end latency, acceptance length, and actual output length on video reasoning tasks. Caption-supported downstream QA could provide an additional check. Under the claimed exact verification, quality checks should validate implementation correctness rather than imply an expected accuracy improvement. End-to-end timing boundaries, repeated-run uncertainty, and baseline configurations need more detail. Clarify inclusion of visual encoding, prefill, routing-key construction, reordering, retrieval, and verification. Evaluation cohorts are small and inference uses one B200; gains for some targets are modest. Several baselines are substantially slower than autoregressive decoding, warranting tuning and bottleneck documentation.

Questions for authors
  1. How does the full method compare with anchor-only and other block-retrieval policies at the same context-access budget and kernel implementation? What are the separate latency gains of prefix aggregation and Hilbert ordering on the longest workloads?

  2. Can the method be evaluated on realistic long-output video reasoning with a compatible target? Please also provide greedy-output consistency checks and describe exact verification at nonzero temperature.

  3. What operations are included in end-to-end timing, and how stable are the improvements across repeated paired measurements? What configurations and tuning were used for the slower speculative baselines?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uurX (1/3)

Q1. Same-budget retrieval policies and component-level latency

On the longest workload (Fig. 3, Video-MME long, 128K), we compare component ablations and a sliding-window baseline using the same kernel and a fixed budget of 2,560 keys per draft step.

SparseDFlash uses the default configuration, w=512, c=32 and K=64, with Hilbert ordering at C=2. Each ablation removes one component. The sliding-window baseline instead reads only the 2,560 most recent tokens.

Retrieval policy (2,560 keys per step)τΔτ vs. OursDecode-only speedup
SparseDFlash (Ours) (prefix-aware, Hilbert C=2)2.401–1.525×
Anchor-only routing2.368−0.033 ± 0.0201.502×
Uniform aggregation2.375−0.026 ± 0.0151.505×
Raster order (no Hilbert)2.381−0.019 ± 0.0141.517×
Sliding window (2,560 recent tokens)2.333−0.068 ± 0.0271.494×
DFlash (full attention, Fig. 3 reference)2.332–

SparseDFlash achieves the highest acceptance and decode-only speedup at the same context-access budget. With per-cycle costs closely matched across policies (33.2–33.5 ms), both prefix-aware aggregation and Hilbert ordering improve the observed acceptance–latency trade-off. Table 2 provides further evidence for prefix-aware aggregation on VDC, where it improves τ over uniform weighting by 0.242 ± 0.028.

The largest acceptance gap is to the sliding window. Table 6 supports the same conclusion across budgets: a smaller window with remote retrieval achieves higher acceptance than a larger recent window at all three context lengths, despite reading fewer keys.

Measurement details. Δτ is the paired per-clip difference from SparseDFlash (Ours), reported as mean ± standard error. Speedups use a shared AR reference measured in this experiment. They should be compared within this table rather than directly with Table 7, whose AR reference was measured in a separate session. We add this comparison to Appendix B.

AuthorsOpenReview ↗

Response to Reviewer uurX (2/3)

W2 / Q2a. Query-driven video reasoning with longer outputs

We evaluate query-driven reasoning with the Qwen3-VL-8B target, DFlash checkpoint and SparseDFlash configuration from Table 1, at T=0.

Video-MME reasoning. We pair each of the 50 Table 1 videos with its original multiple-choice question and ask for detailed reasoning followed by the final answer letter. Prompts contain approximately 47K tokens, with a maximum of 512 output tokens.

MethodAccuracyMean output length (tokens)τEnd-to-end speedup
Autoregressive62.0%157–1.000×
DFlash58.0%1612.6471.235×
SparseDFlash58.0%1612.6131.623×

End-to-end speedup follows Table 1 and is computed from the wall-clock time of the full generation call (Q3a). SparseDFlash produces token-identical outputs to DFlash with a higher end-to-end speedup, and is faster on 48 of the 50 questions. The accuracy difference from AR is shared by DFlash and is discussed in Q2b.

Longer multi-hop answers on MINERVA. In our trials, Qwen3-VL-8B's answers to Video-MME questions remained relatively short, even with alternative reasoning prompts and thinking enabled. We therefore also evaluate MINERVA (Nagrani et al., arXiv:2505.00681, 2025), whose questions combine evidence from several parts of a video. We sample videos with a fixed seed, with one question per video, and evaluate 50 questions on videos ranging from 81 seconds to 1.7 hours.

The prompt asks for an open-ended answer that decomposes the question, cites a timestamp for each sub-question and combines the findings. It requests at least 200 words, with a maximum of 4,096 tokens. Both evaluation prompts appear in full in Appendix D.2.

On some questions, the output collapses into repetition (more than 30% of 8-grams repeat an earlier 8-gram, or the token limit is reached), which inflates acceptance. We skip these questions and report the first 50 questions in sampling order with non-collapsed outputs.

Method (50 questions)Output length (tokens, median)τEnd-to-end speedup vs. AR
DFlash5173.0001.346×
SparseDFlash5173.0121.976×

SparseDFlash maintains DFlash's acceptance length with a higher end-to-end speedup. We add both evaluations to Appendix D.2. Output-consistency checks follow below.

AuthorsOpenReview ↗

Response to Reviewer uurX (3/3)

W3 / Q3a. Timing boundaries

For every method, end-to-end timing uses CUDA-synchronized wall-clock time around the full generation call. The timed region includes visual encoding, target prefill, routing-key construction, reordering, block retrieval, drafting and verification, including acceptance and cache truncation. Only CPU preprocessing (video decoding, frame sampling and tokenization) is cached before timing, and it is identical across methods.

The first clip of each benchmark is used for three warm-up runs of the baseline and every method. End-to-end speedup averages the per-clip throughput ratios to AR decoding. The decode-only speedup in Fig. 3 subtracts target prefill time measured in the AR pass. We add these definitions to Section 5.1.

W3 / Q3b. Measurement stability

Each speedup is a paired per-clip ratio: the throughput of a method divided by that of AR decoding on the same clip, measured in the same run. For the Table 1 measurements of Qwen3-VL-8B, the end-to-end speedup over AR (mean ± standard deviation across clips) is:

TemperatureMethodVDCMVBenchLongVideoBenchVideo-MME
T=0DFlash2.788 ± 0.2552.439 ± 0.3831.126 ± 0.1981.342 ± 0.590
T=0SparseDFlash2.843 ± 0.2402.458 ± 0.2711.823 ± 0.1961.953 ± 0.360
T=1DFlash2.093 ± 0.2061.887 ± 0.2920.878 ± 0.1761.036 ± 0.413
T=1SparseDFlash2.093 ± 0.1881.921 ± 0.1951.499 ± 0.1401.555 ± 0.216

Both methods are evaluated on the same clips. Across all four benchmarks and both temperatures, SparseDFlash achieves the same or higher mean end-to-end speedup, with a lower observed standard deviation across clips.

In three additional runs of the LongVideoBench and Video-MME measurements at T=0, τ and the output length are identical to Table 1 on every clip, and each method's mean end-to-end speedup, computed against the AR reference of Table 1, varies by at most 1.1% across runs.

W3 / Q3c. Baseline configurations and bottlenecks

All baselines use the same harness, clips, prompts, frame and pixel budgets, generation length and AR reference, with batch size 1 on one B200. Target prefill and AR decoding use FlashAttention-2. We port each method from its official code to Qwen3-VL and Qwen3.5, adding GatedDeltaNet state rollback for Qwen3.5.

SpecVLM is training-free. We swept layer skipping, draft-tree depth, and frame and resolution budgets, and Table 1 reports the fastest setting found, which uses a fixed draft tree with 25 nodes, depth 5 and top-k 10. ViSpec and MSD keep the drafter architectures and the EAGLE-2 dynamic draft tree of their papers (30 tokens, depth 3, top-k 8), and are retrained for each target on the same data as DFlash (Section 5.1).

Measured bottleneck. Tree verification requires an explicit attention mask, which the FlashAttention-2 kernel does not accept. The official implementations of the three baselines likewise verify the tree without FlashAttention-2, and we run it with PyTorch scaled dot-product attention. On Qwen3-VL-8B, verifying a 30-token tree rises from 0.92× the cost of one AR step at 2K tokens to 2.60× at 20K. Single-token decoding stays nearly flat over this measured range.

Sequential drafting adds further cost. SpecVLM requires five full-model passes per cycle, while MSD reads the full visual context at every draft step. Together, these costs keep measured speedups near or below 1× on long-video contexts. The baselines do accelerate at shorter contexts. For example, ViSpec reaches 1.77× at 4.5K tokens. We add these configurations and measurements to Appendix C.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (to all reviewers)

We thank all reviewers for their careful reading and constructive feedback. The revision adds:

  1. Query-driven video QA on Video-MME and MINERVA, extending the evaluation beyond generic video description (Reviewers uurX, qiix).
  2. Same-budget retrieval comparisons and component-level speedup measurements on the longest workload (Reviewer uurX).
  3. Clearer positioning against STD, DyCoke, Jenga and DFSAttn (Reviewer JStg).
  4. A latency breakdown for Qwen3.5-9B, identifying the verification bottleneck in GatedDeltaNet (Reviewer JStg).
  5. Timing and baseline details, including measurement boundaries, per-clip variation and baseline configurations (Reviewer uurX).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.