- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SparseDFlash uses sparse attention to speed up long-video speculative decoding. It keeps the full context but selects only relevant blocks for each drafting step. The selection gives more weight to early draft positions. Visual tokens are reordered to group nearby regions across frames. The method adds no learnable parameters and needs no extra training. Experiments on four video datasets show faster generation than full-attention DFlash with the same drafter weights.
Critical strengths
- Dynamic retrieval reduces context access while keeping distant video evidence available.
- The method adds no learnable parameters and requires no extra training. On Qwen3-VL-8B, it runs 1.6–1.7× faster than DFlash on LongVideoBench.
Critical weaknesses
- The novelty over prior work needs clearer justification. STD [1] already uses sparse drafting and dense verification for video understanding. DyCoke [2] allows visual KV entries to be reactivated during decoding. Jenga [3] and DFSAttn [4] already use 3D Hilbert reordering for video sparse attention. The paper should clarify what its parallel drafting and prefix-aware routing add to these designs.
- The speedup on Qwen3.5-9B is small. At temperature 1, the average speedup over autoregressive decoding is only 1.049×. The authors attribute this to verification overhead in the GatedDeltaNet layers. However, no latency breakdown supports this explanation. More detailed measurements would help identify the main bottleneck.
[1] Xuan Zhang et al. Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs. ACL, 2025.
[2] Keda Tao et al. DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models. CVPR, 2025.
[3] Yuechen Zhang et al. Training-Free Efficient Video Generation via Dynamic Token Carving. NeurIPS, 2025.
[4] Jie Hu et al. DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation. arXiv:2605.23445
Questions for authors
- What is the main technical advance over STD, DyCoke, and prior Hilbert-based sparse attention methods?
- How much latency comes from drafting and verification on Qwen3.5-9B? Do these measurements support the explanation for its limited speedup?
3 comments
Response to Reviewer JStg (2/2)
W2 / Q2. Qwen3.5-9B latency breakdown
On Qwen3.5-9B, verifying a 16-token block costs about two AR steps. GatedDeltaNet accounts for almost all of the additional cost.
We profile vanilla DFlash on the same 50 VDC clips, using approximately 9K-token inputs, T=0 and up to 512 generated tokens under the Table 1 configuration. We synchronize around each phase and use CUDA events for timings by layer type. These instrumented measurements are reported separately from the end-to-end results.
All timings below are in milliseconds, and τ is tokens advanced per cycle. “Cycle” denotes DFlash, and “AR” denotes one autoregressive step.
| Phase / metric | Qwen3-VL-8B: cycle | Qwen3-VL-8B: AR | Qwen3.5-9B: cycle | Qwen3.5-9B: AR |
|---|---|---|---|---|
| Drafting | 3.2 | 5.8 | ||
| Target forward | 22.5 | 22.6 | 62.8 | 31.2 |
| ├ GatedDeltaNet | – | – | 47.1 | 16.0 |
| ├ Attention | 15.2 | 13.4 | 4.6 | 4.6 |
| ├ MLP | 3.0 | 3.5 | 3.4 | 3.3 |
| └ Embeddings, norms and LM head | 4.3 | 7.7 | ||
| Cache / state restoration | 0.1 | 0.9 | ||
| Other: sampling, acceptance and bookkeeping | 0.9 | 1.0 | ||
| Total per cycle | 26.7 | 70.5 | ||
| τ | 3.65 | 2.77 |
Attention and MLP remain close to their single-step costs on both targets. GatedDeltaNet, however, costs 2.94× a single step during verification. Our verifier updates the recurrent state token by token so it can restore any accepted prefix. Restoration itself is inexpensive. We observe the same pattern at T=1, where GatedDeltaNet verification costs 2.8× a single step.
This approximately 9K-token VDC profile identifies target verification as the dominant cost on the hybrid model. SparseDFlash reduces the draft-side context-access cost, with Table 1 showing improvements in end-to-end speedup of about 9–11% over DFlash on the longer benchmarks at T=0.
Verification-side optimizations. HybridSpec (Wang et al., arXiv:2611.04055, Nov. 2026) addresses the same bottleneck on hybrid GatedDeltaNet-MoE targets through adaptive verification lengths, parallel processing of the selected prefix and efficient state reconstruction. These verification-side optimizations are potentially complementary to SparseDFlash's reduction of drafting cost. We add the breakdown to Appendix E and discuss the verification bottleneck in the revision.
Thanks for taking a close look at how this work differs from prior methods. The new latency breakdown also clears up my second concern. It shows that most of the time is spent on target verification, with GatedDeltaNet accounting for nearly all of the extra forward-pass cost. This helps explain why speeding up the drafter gives only a small overall gain on Qwen3.5-9B. I’m satisfied with this explanation and have decided to raise my score from 2 to 3.
Response to Reviewer JStg (1/2)
We thank the reviewer for the careful review and the pointers to related work. We add all four papers to Related Work.
W1 / Q1. Technical advance over STD, DyCoke and Hilbert-based sparse attention
Main advance. SparseDFlash adapts dynamic block retrieval to a frozen parallel drafter, with selection matched to prefix acceptance. A shared block set must support jointly predicted positions, but a later candidate can be accepted only if all earlier candidates are accepted. We therefore compute relevance separately for each position and aggregate it with exponentially decaying prefix weights (Section 4.2).
Within SparseDFlash, replacing these weights with uniform position weights lowers τ by 0.242 on VDC (Table 2). Tables 4–5 demonstrate applicability to official DFlash checkpoints without additional training.