- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes InstaGen, a training-free sparse attention method for accelerating video generation while preserving generation quality. Building on SVG2's semantic-aware query/key clustering, InstaGen introduces a group-shared importance sampling strategy that deterministically selects high-importance key blocks and samples the remaining keys according to their estimated attention importance. The sampled contributions are reweighted to approximate dense attention, and the method is implemented with a GPU-efficient fused attention kernel.
The authors evaluate InstaGen on Wan2.1-T2V-1.3B and 14B, comparing it against SVG, SVG2, and SVG-EAR. The experiments show improved attention approximation accuracy and generation quality at matched attention density and latency, with end-to-end speedups over dense attention.
Critical strengths
The experiments demonstrate substantial improvements over existing sparse attention methods.
Critical weaknesses
The proposed method builds heavily on existing work. In particular, the query/key clustering and semantic-aware permutation are inherited from SVG2, while the idea of combining deterministic token selection with sampling-based residual attention approximation has already been explored in vAttention. The main contribution appears to be integrating importance sampling into an existing sparse attention framework. Although the empirical results are encouraging, I find the algorithmic novelty relatively incremental, and it is unclear whether the proposed combination constitutes a sufficiently significant methodological advance.
Questions for authors
Could the authors clarify the key methodological novelty compared with SVG2 and vAttention? In particular, beyond combining SVG2's clustering framework with sampling-based residual attention approximation, what is the fundamental algorithmic contribution of InstaGen? Why this advancement is important?
4 comments
More instantiations of InstaGen: InstaGen improves over different fixed sparse attention methods
| Method | PSNR ↑ | LPIPS ↓ | SSIM ↑ | Density ↓ |
|---|---|---|---|---|
| SVG2 | 26.11 | 0.122 | 0.834 | 48.92% |
| InstaGen (importance, 8+8, warmup), SVG2's clustering | 27.69 | 0.071 | 0.869 | 41.20% |
| SVOO | 28.30 | 0.081 | 0.908 | 41.57% |
| InstaGen (importance, 8+8, warmup), SVOO's clustering | 30.83 | 0.047 | 0.928 | 42.98% |
| Rectified SpaAttn | 22.66 | 0.187 | 0.801 | 37.12% |
| InstaGen (importance, 8+8, warmup), Rectified's blocks | 27.12 | 0.087 | 0.874 | 30.33% |
For Wan2.1-T2V-1.3B, 720p, 81 frames, we observe improvements with InstaGen over methods that use clustering schemes other than SVG2, including one that does not use clustering at all. Thus, InstaGen is not a modification of SVG2, but rather provides a better estimator for residual attention that can be applied to other sparse attention methods.
More ablations on dense warmup steps: InstaGen does not need any dense steps
Although the paper already demonstrates this trend (Table 1), we provide additional InstaGen instantiations to establish its generality.
| Method | PSNR ↑ | LPIPS ↓ | SSIM ↑ | Density ↓ |
|---|---|---|---|---|
| SVG2 | 26.11 | 0.122 | 0.834 | 48.92% |
| InstaGen (importance, 16+16, no warmup), SVG2's clustering | 27.72 | 0.061 | 0.891 | 32.00% |
| InstaGen (uniform, 16+16, no warmup), SVG2's clustering | 26.54 | 0.088 | 0.876 | 32.00% |
| SVOO | 28.30 | 0.081 | 0.908 | 41.57% |
| InstaGen (importance, 16+16, no warmup), SVOO's clustering | 30.94 | 0.041 | 0.935 | 32.00% |
| InstaGen (uniform, 16+16, no warmup), SVOO's clustering | 28.20 | 0.073 | 0.906 | 32.00% |
| Rectified SpaAttn | 22.66 | 0.187 | 0.801 | 37.12% |
| InstaGen (importance, 16+16, no warmup), Rectified's blocks | 26.99 | 0.085 | 0.886 | 32.08% |
| InstaGen (uniform, 16+16, no warmup), Rectified's blocks | 27.61 | 0.077 | 0.896 | 32.08% |
Across all three clustering or block layouts, importance-based InstaGen improves reconstruction quality while reducing density to approximately 32%. The uniform variants remain competitive, indicating that the benefits are not tied to a particular clustering scheme.
Eliminating warmup matters most for long videos. A method that devotes 30% of its denoising steps to dense warmup has a hard latency floor of 30% of the dense baseline, limiting its theoretical speedup to 3.33×.
| Experiment | Frames | Method | Latency | Speedup |
|---|---|---|---|---|
| Dense per-step scaling | 81 | Dense attention | 4.6 s/step | — |
| Dense per-step scaling | 561 | Dense attention | 194.4 s/step | — |
| Matched-quality frame scaling | 561 | Dense attention | 162.0 min | 1.00× |
| Matched-quality frame scaling | 561 | SVG2 | 82.8 min | 1.97× |
| Matched-quality frame scaling | 561 | InstaGen with warmup | 61.9 min | 2.62× |
| Matched-quality frame scaling | 561 | InstaGen without warmup | 57.3 min | 2.83× |
On Wan-1.3B, increasing the video length from 81 to 561 frames raises dense-attention latency from 4.6 to 194.4 seconds per step. In the matched-quality experiment at 561 frames, InstaGen without warmup reduces denoising time from 162.0 to 57.3 minutes, yielding a 2.83× speedup on a single H100. We expect the advantage over warmup-based methods to grow further as video length increases.
Thank you for the clarification. I appreciate the additional experiments demonstrating that InstaGen can generalize to different sparse attention methods. This provides stronger evidence that InstaGen is a generalizable framework rather than an optimization specific to SVG2. Considering this additional evidence and the engineering contributions, I am willing to raise my score to 2.
However, I still have concerns about its methodological novelty, as the core ideas build substantially on prior work. Overall, while I would not object to the paper being accepted, I am not inclined to recommend its acceptance given my remaining concerns about novelty.
Thank you for raising your score and for recognizing that InstaGen generalizes beyond SVG2. We would like to address the remaining concern directly, because we see the core contribution as methodological rather than an engineering detail.
InstaGen takes well-known statistical tools, such as top-k selection, importance sampling and Horvitz–Thompson weighting, and combines them to estimate the residual attention that sparse selection methods skip. More importantly, InstaGen goes beyond these tools to improve the state of the art in sparse video generation:
- The sampling structure is a finding, not a given. On full self-attention, both direct options fail. Per-query sampling, the form vAttention uses for decoding, cannot use coalesced memory access and ran slower than full attention in our experiments. Sampling 2D blocks of the attention map restores coalesced access but compromises quality significantly. InstaGen shares one draw across a query group but keeps individual keys, so samples are correlated only along the query axis and never along the key axis.
- Removal of dense warmup. Every method we compare with needs dense warm-up steps. InstaGen runs every step sparse and still outperforms SVG2, SVOO and Rectified SpaAttn run with their warm-up on Wan2.1-1.3B, and SVG-EAR on 100 prompts at lower density. Since warm-up caps the speedup (3.3× for a 30% warm-up), this matters most for long videos.
- It improves every base it is plugged into, as you note, including Rectified SpaAttn, which uses no clustering at all.
Recent methods in this area, such as SVOO and SVG-EAR, also build on SVG2's framework, and are judged by what they change. InstaGen changes how the skipped attention is treated, which, to our knowledge, no prior video method does stochastically. We hope this clarifies why we see it as a methodological advance, and we thank you again for the constructive discussion.
Response
We thank the reviewer for their time. We believe that the review may not fully capture the scope of InstaGen's contributions, particularly in its relationship to the advances in SVG2 and prior ideas from vAttention. We would like to clarify that InstaGen goes beyond directly building upon these existing ideas, and that this interpretation underestimates the contributions of our work. We elaborate on this in the following points.
First, InstaGen is not dependent on SVG2. It can work with any existing sparse attention mechanism. The fact that InstaGen is parameterized by existing methods is not a drawback but an asset, as it allows new advances in sparse attention to be readily integrated into InstaGen. More importantly, InstaGen goes beyond simply leveraging existing methods: it addresses a key limitation of these sparse attention methods by performing residual estimation, helping achieve a better quality-efficiency tradeoff where existing methods often struggle. The gains from residual estimation are already evident in the paper. To further support this, we have added experiments combining InstaGen with SVOO and Rectified Sparse Attention.
Second, while vAttention does discuss residual estimation and a verified algorithm for it, extending the full vAttention algorithm to prefill workloads is not straightforward. Even if we borrow the idea of uniform sampling, uniform sampling independently per query cannot be efficiently accelerated on GPUs due to the lack of coalesced memory access. A straightforward extension using 2D block sampling to enable coalesced access, on the other hand, compromises quality significantly. Our contribution is in identifying the appropriate sampling structure and determining when to sample to make it work efficiently for prefill workloads. The resulting speedups and quality improvements demonstrate the value of these design choices.
More importantly, InstaGen goes beyond these ideas to improve the state of the art in sparse video generation:
We extend residual estimation to importance sampling, further leveraging the underlying black-box sparse attention mechanism. This improves quality while maintaining comparable execution speed, enabling improvements in the quality-efficiency Pareto frontier.
We want to emphasize that, to the best of our knowledge, we are the first to achieve fully sparse diffusion while simultaneously improving generation quality. Previous methods require dense diffusion during their initial steps. This is an important distinction in the capabilities enabled by InstaGen, which can help unlock long-video generation. (See experiments below.)
For both uniform and importance sampling, we enable GPU-friendly sampling granularity which strikes a balance between efficiency and quality of estimation. This avoids uncoalesced memory access by avoiding sampling independently per query. On the other hand, it introduces correlations only along the query axis while avoiding correlations along the key axis, which is catastrophic for estimation.
To the best of our knowledge, we are the first to introduce stochastic residual estimation for sparse video generation, improving upon the deterministic residual estimation used in prior methods such as SVG-EAR (one centroid key per skipped cluster) and Rectified SpaAttn (pooled query-key scores).
Below, we provide additional empirical evidence to support our claims beyond what is already presented in the paper.