acceptodds
Withdrawn from ICLR 2027

AR: Training-Free Sparse Attention for Autoregressive Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Autoregressive video diffusion has recently emerged as a promising paradigm for real-time and streaming video generation, owing to its causal decoding formulation and natural support for key–value (KV) caching. However, as generation proceeds, the KV cache grows monotonically over rollout, making attention over long KV histories the dominant inference bottleneck. In this work, we propose \textbf{A^2R} (nchored and daptive eusable Attention), a training-free sparse attention framework for long-horizon autoregressive video diffusion. Our design is motivated by three observations: early chunks act as persistent visual anchors for later generation, attention sparsity varies substantially across heads and routing patterns remain stable across intermediate denoising steps. Based on these observations, AR combines an anchored dense-to-sparse transition, per-head adaptive block routing and reusable sparse patterns that are refreshed only at the final denoising step. We further implement an exact Triton-based sparse attention kernel that computes attention only over retained KV blocks. Extensive experiments on autoregressive video diffusion backbones show that AR achieves around 77% average attention sparsity, reduces attention runtime by \textbf{2.1\times}, and delivers a \textbf{1.33\times} end-to-end speedup while preserving strong visual quality and temporal consistency, yielding a favorable quality–efficiency trade-off for video generation.

Primary area:
generative models
Keywords:
video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GbLSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the continuously growing attention overhead brought by KV‑cache in long‑horizon generation of autoregressive video diffusion models, and proposes A²R (Anchored and Adaptive Reusable Attention), a training‑free sparse‑attention framework. It consists of three core components: (1) Anchored dense‑to‑sparse transition: the first video chunk is kept as a dense visual anchor to avoid corruption of scene layout and object identity; (2) Per‑head adaptive block routing: dynamically selects KV blocks to retain based on a relevance threshold instead of adopting a fixed Top‑K sparsity ratio; (3) Reusable sparse routing patterns across denoising steps: routing is only refreshed at the final denoising step to reduce computation, leveraging the stable routing structure observed among intermediate denoising steps. The authors conduct experiments on autoregressive video diffusion models and demonstrate that A²R substantially reduces inference overhead for long‑horizon video generation while preserving generation quality.

Critical strengths

This paper identifies two unique properties of autoregressive video diffusion: head‑wise sparsity heterogeneity and stable routing across intermediate denoising steps. These serve as key insights distinguishing this work from sparse‑attention solutions originally designed for LLMs. The motivations are well‑aligned with the chunk‑wise rollout nature of autoregressive video generation, rather than a naive transfer of existing LLM sparse‑attention techniques to video diffusion.

The entire framework is training‑free and operates purely at inference time without requiring model fine‑tuning. Equipped with an exact Triton‑based sparse‑attention kernel, the attention computation itself introduces no approximation even when pruning KV blocks. It achieves considerable reduction of KV‑cache overhead for long sequences with manageable degradation in video quality.

Critical weaknesses

Key quality evaluation metrics could be further strengthened: the full set of VBench metrics is not reported, and additional benchmarks could be supplemented.

Questions for authors
  1. How are finetune‑aware sparse baselines such as SLA implemented in your experimental comparisons?
  2. Could you add experimental results on more targeted benchmarks such as VBench‑long?
Reviewer_RVf9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a training-free sparse-attention framework for autoregressive video diffusion. It combines dense attention for the initial chunk, adaptive block routing based on cumulative relevance mass, and reuse of routing patterns across intermediate steps with a final-step refresh. The reported experiments show reduced inference latency while maintaining generation quality close to dense attention.

Critical strengths
  • The method provides measurable end-to-end improvements without additional training. This makes the approach potentially useful as an inference-time optimization.
  • Tables 2–4 provide useful ablations examining the contributions of dense initialization, adaptive routing, and routing reuse.
Critical weaknesses
  • The paper has limited novelty. Block-sparse attention for video diffusion has been explored in VSA and VMoBA, cross-step pattern reuse was introduced in AdaSpa, and dense first-chunk initialization was used in Light Forcing. A2RA^2R primarily integrates these established ideas into a training-free autoregressive framework. Although implementation details differ, the paper does not clearly establish a substantive new mechanism or autoregressive-specific insight beyond this combination. This is my main concern.

  • Related to this, the comparisons do not sufficiently isolate the proposed method’s contribution relative to closely related approaches. Despite including several sparse-attention baselines, the paper does not directly compare against VSA, VMoBA, or AdaSpa. The static per-head and per-(head,layer) controls in Table 9 are useful ablations, but they are not substitutes for existing input-adaptive routing methods.

  • The incremental gains over the closest reported autoregressive baseline are modest. Compared with the Light Forcing adaptation, A2RA^2R reduces Self-Forcing latency from 4.66 to 4.59 seconds. On LongLive, latency decreases from 5.37 to 5.12 seconds.

  • There is also a minor presentation issue. Many citation groups in the introduction and related work are rendered as narrative author–year citations where parenthetical citations would be more appropriate. These should be corrected using the appropriate citation commands.

Questions for authors
  • What specific methodological advance does A2RA^2R offer beyond combining existing cumulative-mass block routing, dense first-chunk initialization, and cross-step sparse-pattern reuse?
Reviewer_FWKkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents A²R, a training-free sparse-attention framework for autoregressive video diffusion. The method generates the initial chunk with dense attention, constructs head- and query-block-dependent sparse supports using cumulative-mass selection over mean-pooled query/key representations, and reuses these supports across intermediate model evaluations while refreshing them at the final evaluation. A custom Triton kernel computes attention on the original tokens within the retained support. On Self-Forcing and LongLive, the paper reports approximately 1.3× end-to-end acceleration with small decreases in aggregate VBench scores. The evaluation also includes component ablations, longer-rollout experiments, kernel profiling, and composition with FP8 inference.

Critical strengths
  1. The paper reports meaningful end-to-end improvements on two autoregressive backbones: latency decreases from 6.11 to 4.59 seconds on Self-Forcing and from 6.65 to 5.12 seconds on LongLive, while VBench Total decreases by only 0.2 and 0.1 points, respectively. The reported motion and consistency dimensions also remain close to the dense baselines. These results suggest that the method could be useful for accelerating existing models without retraining.

  2. The paper evaluates dense initialization, adaptive routing, output reuse versus support reuse, and different refresh schedules. For example, routing reuse reduces latency from 4.83 to 4.59 seconds with a small reported score change, while refreshing the final evaluation improves Total from 83.0 to 83.8 relative to reusing the initial support throughout. These controls provide a useful explanation of the implementation choices.

Critical weaknesses
  1. The pooled-Q/K cumulative-mass router in Equations (4)–(10) substantially overlaps the mean-pooling and TopCdf mechanism in SpargeAttention. Consequently, grouping that method with fixed-top-K or globally uniform sparsity approaches does not accurately characterize the relevant prior mechanism. Light Forcing also explicitly keeps the first chunk dense and studies the downstream degradation caused by sparsifying it. Furthermore, heterogeneous head sparsity and cross-step sparse-mask reuse have been investigated in AdaSpa. A²R may offer a useful integration for autoregressive video diffusion, particularly through its final cache-update refresh policy, but the submission needs to distinguish this specific contribution from the existing ingredients and demonstrate its incremental value under controlled conditions.

  2. Relative to LF-attn, the reported latency improvements are approximately 1.5% on Self-Forcing and 4.7% on LongLive, with Total improvements of 0.3 and 0.2 points. These are encouraging but modest margins, and the paper does not specify sample counts, seed pairing, quality uncertainty, repeated timing variability, or sufficiently detailed baseline tuning. The main table also labels the comparison as Light Forcing, whereas Appendix C.1 identifies it as an attention-only adaptation. In addition, Table 6 reports A²R quality up to 30 seconds but omits dense quality at the corresponding durations. It therefore does not isolate the additional degradation attributable to sparsification as rollout length increases. These omissions limit the evidence for the claimed quality–efficiency advantage in the setting that motivates the paper.

Questions for authors

Contribution and controlled comparison. What algorithmic differences distinguish Equations (4)–(10) from SpargeAttention’s pooled-Q/K cumulative-mass routing, and the dense initialization policy from that used in Light Forcing? Could the authors compare an adapted cumulative-mass router under the same kernel and dense-initialization policy, varying the reuse/refresh schedule at matched operating points? The existing no-reuse ablation is useful, but a direct connection to the relevant prior mechanisms would help establish the specific contribution.

  1. Which Light Forcing components, weights, and hyperparameters are retained in LF-attn? Please provide the number of prompts and samples, the seed-pairing protocol, repeated latency measurements, and uncertainty estimates for the quality differences. Does A²R retain an advantage over LF-attn across matched-quality or matched-latency operating points?

  2. What are the dense-baseline quality scores at every duration in Table 6 under the same evaluation protocol? Please specify the attention window, frame-sink policy, and cache configuration. How much additional degradation does A²R introduce in later video segments, particularly for subject consistency and temporal coherence?

  3. Does the final evaluation correspond to a separate clean-context KV-update pass rather than another denoising update? Please specify the actual timestep schedule. Also, what is the denominator and averaging procedure for achieved sparsity, and does Figure 9 show executed sparsity or candidate routing statistics? The nonzero sparsity shown for chunk 0 appears inconsistent with the stated dense-initialization policy unless a different indexing or accounting convention is used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.