acceptodds
Under review as a conference paper at ICLR 2027

InstaGen: Group-Shared Importance Sampling for Fast Video Generation

Abstract

Diffusion Transformers (DiTs) are the dominant architecture for video generation but suffer from significant latency due to the quadratic complexity of attention. Block-sparse attention reduces this cost by computing only the top-k attention blocks per query, or top-p, which varies k to meet a target attention-mass threshold, and skipping the rest. Achieving high generation quality under a fixed block budget requires solving two problems: (1) Top-k block selection: the blocks carrying the most attention mass must be identified accurately for each query cluster, but existing methods rank blocks by the score of a single cluster centroid, which misranks them whenever the queries within a cluster disagree. (2) Skipped block recovery: the blocks below the top-k boundary still carry attention mass, and existing methods either drop them, incurring information loss that grows as k shrinks, or compensate for them with per-block key/value centroids, whose reconstruction error is determined by the clustering and does not decrease as more compute is spent. In this paper, we propose InstaGen, which addresses both. For selection, InstaGen product-quantizes queries and keys so that block scores are cheap enough to compute for every query rather than only the centroid, and each query votes for its top-k blocks; the cluster attends to the blocks with the most votes. For recovery, InstaGen draws a small random sample of tokens from the blocks below the top-k boundary, shared across all queries in the cluster, and computes them together with the top-k blocks in a single attention kernel, with sampled scores reweighted to account for the full skipped region. The sample size is a direct budget: reconstruction error decreases as more tokens are sampled, at a cost that is a small fraction of the top-k computation. InstaGen is training-free and requires no auxiliary predictor. On Wan2.1-1.3B and Wan2.1-14B, InstaGen improves PSNR over SVG2 by 2.9 dB and 1.2 dB at matched attention density and by 1.8 dB and 0.9 dB at matched wall-clock time, and reaches SVG2's best quality 1.13× and 1.20× faster, respectively, while continuing to improve at densities where SVG2 has plateaued.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
video generation, diffusion transformers, block-sparse attention, sampling, raining-free, efficient inference, attention estimation, product quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mQYXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InstaGen, a training-free sparse attention method for accelerating video generation while preserving generation quality. Building on SVG2's semantic-aware query/key clustering, InstaGen introduces a group-shared importance sampling strategy that deterministically selects high-importance key blocks and samples the remaining keys according to their estimated attention importance. The sampled contributions are reweighted to approximate dense attention, and the method is implemented with a GPU-efficient fused attention kernel.

The authors evaluate InstaGen on Wan2.1-T2V-1.3B and 14B, comparing it against SVG, SVG2, and SVG-EAR. The experiments show improved attention approximation accuracy and generation quality at matched attention density and latency, with end-to-end speedups over dense attention.

Critical strengths

The experiments demonstrate substantial improvements over existing sparse attention methods.

Critical weaknesses

The proposed method builds heavily on existing work. In particular, the query/key clustering and semantic-aware permutation are inherited from SVG2, while the idea of combining deterministic token selection with sampling-based residual attention approximation has already been explored in vAttention. The main contribution appears to be integrating importance sampling into an existing sparse attention framework. Although the empirical results are encouraging, I find the algorithmic novelty relatively incremental, and it is unclear whether the proposed combination constitutes a sufficiently significant methodological advance.

Questions for authors

Could the authors clarify the key methodological novelty compared with SVG2 and vAttention? In particular, beyond combining SVG2's clustering framework with sampling-based residual attention approximation, what is the fundamental algorithmic contribution of InstaGen? Why this advancement is important?

4 comments

We thank the reviewer for their time. We believe that the review may not fully capture the scope of InstaGen's contributions, particularly in its relationship to the advances in SVG2 and prior ideas from vAttention. We would like to clarify that InstaGen goes beyond directly building upon these existing ideas, and that this interpretation underestimates the contributions of our work. We elaborate on this in the following points.

First, InstaGen is not dependent on SVG2. It can work with any existing sparse attention mechanism. The fact that InstaGen is parameterized by existing methods is not a drawback but an asset, as it allows new advances in sparse attention to be readily integrated into InstaGen. More importantly, InstaGen goes beyond simply leveraging existing methods: it addresses a key limitation of these sparse attention methods by performing residual estimation, helping achieve a better quality-efficiency tradeoff where existing methods often struggle. The gains from residual estimation are already evident in the paper. To further support this, we have added experiments combining InstaGen with SVOO and Rectified Sparse Attention.

Second, while vAttention does discuss residual estimation and a verified algorithm for it, extending the full vAttention algorithm to prefill workloads is not straightforward. Even if we borrow the idea of uniform sampling, uniform sampling independently per query cannot be efficiently accelerated on GPUs due to the lack of coalesced memory access. A straightforward extension using 2D block sampling to enable coalesced access, on the other hand, compromises quality significantly. Our contribution is in identifying the appropriate sampling structure and determining when to sample to make it work efficiently for prefill workloads. The resulting speedups and quality improvements demonstrate the value of these design choices.

More importantly, InstaGen goes beyond these ideas to improve the state of the art in sparse video generation:

  1. We extend residual estimation to importance sampling, further leveraging the underlying black-box sparse attention mechanism. This improves quality while maintaining comparable execution speed, enabling improvements in the quality-efficiency Pareto frontier.

  2. We want to emphasize that, to the best of our knowledge, we are the first to achieve fully sparse diffusion while simultaneously improving generation quality. Previous methods require dense diffusion during their initial steps. This is an important distinction in the capabilities enabled by InstaGen, which can help unlock long-video generation. (See experiments below.)

  3. For both uniform and importance sampling, we enable GPU-friendly sampling granularity which strikes a balance between efficiency and quality of estimation. This avoids uncoalesced memory access by avoiding sampling independently per query. On the other hand, it introduces correlations only along the query axis while avoiding correlations along the key axis, which is catastrophic for estimation.

  4. To the best of our knowledge, we are the first to introduce stochastic residual estimation for sparse video generation, improving upon the deterministic residual estimation used in prior methods such as SVG-EAR (one centroid key per skipped cluster) and Rectified SpaAttn (pooled query-key scores).

Below, we provide additional empirical evidence to support our claims beyond what is already presented in the paper.

More instantiations of InstaGen: InstaGen improves over different fixed sparse attention methods

MethodPSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG226.110.1220.83448.92%
InstaGen (importance, 8+8, warmup), SVG2's clustering27.690.0710.86941.20%
SVOO28.300.0810.90841.57%
InstaGen (importance, 8+8, warmup), SVOO's clustering30.830.0470.92842.98%
Rectified SpaAttn22.660.1870.80137.12%
InstaGen (importance, 8+8, warmup), Rectified's blocks27.120.0870.87430.33%

For Wan2.1-T2V-1.3B, 720p, 81 frames, we observe improvements with InstaGen over methods that use clustering schemes other than SVG2, including one that does not use clustering at all. Thus, InstaGen is not a modification of SVG2, but rather provides a better estimator for residual attention that can be applied to other sparse attention methods.

More ablations on dense warmup steps: InstaGen does not need any dense steps

Although the paper already demonstrates this trend (Table 1), we provide additional InstaGen instantiations to establish its generality.

MethodPSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG226.110.1220.83448.92%
InstaGen (importance, 16+16, no warmup), SVG2's clustering27.720.0610.89132.00%
InstaGen (uniform, 16+16, no warmup), SVG2's clustering26.540.0880.87632.00%
SVOO28.300.0810.90841.57%
InstaGen (importance, 16+16, no warmup), SVOO's clustering30.940.0410.93532.00%
InstaGen (uniform, 16+16, no warmup), SVOO's clustering28.200.0730.90632.00%
Rectified SpaAttn22.660.1870.80137.12%
InstaGen (importance, 16+16, no warmup), Rectified's blocks26.990.0850.88632.08%
InstaGen (uniform, 16+16, no warmup), Rectified's blocks27.610.0770.89632.08%

Across all three clustering or block layouts, importance-based InstaGen improves reconstruction quality while reducing density to approximately 32%. The uniform variants remain competitive, indicating that the benefits are not tied to a particular clustering scheme.

Eliminating warmup matters most for long videos. A method that devotes 30% of its denoising steps to dense warmup has a hard latency floor of 30% of the dense baseline, limiting its theoretical speedup to 3.33×.

ExperimentFramesMethodLatencySpeedup
Dense per-step scaling81Dense attention4.6 s/step—
Dense per-step scaling561Dense attention194.4 s/step—
Matched-quality frame scaling561Dense attention162.0 min1.00×
Matched-quality frame scaling561SVG282.8 min1.97×
Matched-quality frame scaling561InstaGen with warmup61.9 min2.62×
Matched-quality frame scaling561InstaGen without warmup57.3 min2.83×

On Wan-1.3B, increasing the video length from 81 to 561 frames raises dense-attention latency from 4.6 to 194.4 seconds per step. In the matched-quality experiment at 561 frames, InstaGen without warmup reduces denoising time from 162.0 to 57.3 minutes, yielding a 2.83× speedup on a single H100. We expect the advantage over warmup-based methods to grow further as video length increases.

Thank you for the clarification. I appreciate the additional experiments demonstrating that InstaGen can generalize to different sparse attention methods. This provides stronger evidence that InstaGen is a generalizable framework rather than an optimization specific to SVG2. Considering this additional evidence and the engineering contributions, I am willing to raise my score to 2.

However, I still have concerns about its methodological novelty, as the core ideas build substantially on prior work. Overall, while I would not object to the paper being accepted, I am not inclined to recommend its acceptance given my remaining concerns about novelty.

Thank you for raising your score and for recognizing that InstaGen generalizes beyond SVG2. We would like to address the remaining concern directly, because we see the core contribution as methodological rather than an engineering detail.

InstaGen takes well-known statistical tools, such as top-k selection, importance sampling and Horvitz–Thompson weighting, and combines them to estimate the residual attention that sparse selection methods skip. More importantly, InstaGen goes beyond these tools to improve the state of the art in sparse video generation:

  1. The sampling structure is a finding, not a given. On full self-attention, both direct options fail. Per-query sampling, the form vAttention uses for decoding, cannot use coalesced memory access and ran slower than full attention in our experiments. Sampling 2D blocks of the attention map restores coalesced access but compromises quality significantly. InstaGen shares one draw across a query group but keeps individual keys, so samples are correlated only along the query axis and never along the key axis.
  2. Removal of dense warmup. Every method we compare with needs dense warm-up steps. InstaGen runs every step sparse and still outperforms SVG2, SVOO and Rectified SpaAttn run with their warm-up on Wan2.1-1.3B, and SVG-EAR on 100 prompts at lower density. Since warm-up caps the speedup (3.3× for a 30% warm-up), this matters most for long videos.
  3. It improves every base it is plugged into, as you note, including Rectified SpaAttn, which uses no clustering at all.

Recent methods in this area, such as SVOO and SVG-EAR, also build on SVG2's framework, and are judged by what they change. InstaGen changes how the skipped attention is treated, which, to our knowledge, no prior video method does stochastically. We hope this clarifies why we see it as a methodological advance, and we thank you again for the constructive discussion.

Reviewer_SJJWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InstaGen is a training-free method for accelerating video diffusion transformers. It computes attention exactly for selected high-importance keys and estimates the remaining contribution through importance sampling. Queries in the same cluster share the sampled keys to support efficient GPU execution. Experiments on Wan2.1-T2V-1.3B and 14B show improved fidelity to dense-attention outputs compared with existing sparse methods.

Critical strengths

(1) The method requires no additional training or fine-tuning, making it practical for accelerating existing video generation models.

(2) The reported results show that InstaGen can preserve dense-generation outputs better at a matched attention budget. For example, at 44.3% overall attention density on Wan2.1-T2V-1.3B, it improves PSNR from 25.74 to 29.42 dB and reduces LPIPS from 0.131 to 0.064 compared with SVG2. This is a meaningful improvement in fidelity while reducing computation.

Critical weaknesses

(1) The main quality evaluation uses only 22 prompts, with one generation per prompt. These prompts were selected for relatively high motion from 11 VBench prompt lists, which provides a useful targeted test, but leaves uncertainty about performance on a broader range of inputs. Only four VBench dimensions are reported, so other aspects of video quality remain insufficiently evaluated. A larger, more diverse prompt set and broader VBench coverage would make the quality-preservation claims more convincing.

(2) The benefit and accuracy impact of group-shared sampling are not directly measured. The paper reports end-to-end speedups and a latency breakdown, but does not compare shared sampling with independent per-query sampling under controlled settings. It is therefore unclear how much efficiency comes specifically from sharing samples and whether this introduces an accuracy cost. The experiments with different cluster counts are informative, but also change the importance estimates and do not isolate the effect of sharing.

Questions for authors

(1) How sensitive is the method to the allocation between exact computation and sampling? Please compare different E/S splits, such as 24+8, 16+16, and 8+24, while keeping the total budget and other settings fixed. The current experiments mainly vary the total budget with E:S fixed at 1:1.

(2) Can you evaluate a larger and more diverse prompt set—for example, 100 or more prompts—and report additional VBench dimensions? An aggregate score from a complete VBench evaluation and results over repeated runs would also help assess overall quality and stability.

(3) How much efficiency is gained specifically from group-shared sampling, and does sharing affect accuracy? A controlled comparison with independent per-query sampling, using the same importance scores and sampling budget, would be helpful. Please report sampling time, attention-kernel time, and approximation error.

(4) Do the improvements extend beyond the Wan2.1 family? Additional results on another video generation model, such as HunyuanVideo, would help establish the method’s broader applicability.

4 comments

We thank the reviewer for their time and effort. We address the raised questions below,

Ablations on Budget Allocation

We report ablations for uniform and importance sampling on Wan2.1-T2V-1.3B, evaluating uniform sampling both with and without warmup. Each split denotes how the fixed budget is divided between exact and sampled keys; for example, 0.9/0.1 allocates 90% to exact keys and 10% to sampled keys. Performance is strongest and remains stable when a larger share of the budget is allocated to exact keys. Notably, even a 0.5/0.5 split with uniform sampling outperforms the base algorithm, demonstrating robustness to the precise budget allocation.

Uniform sampling ablation

1. With dense warmup

Method (exact/sampled)PSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG2 (top-p 0.9), warmup29.610.1150.91844.79%
InstaGen (uniform, 1.00 / 0.00, warmup)27.150.1850.87544.84%
InstaGen (uniform, 0.90 / 0.10, warmup)31.830.0630.94544.84%
InstaGen (uniform, 0.75 / 0.25, warmup)31.510.0580.94344.84%
InstaGen (uniform, 0.50 / 0.50, warmup)30.710.0820.92944.84%
InstaGen (uniform, 0.25 / 0.75, warmup)29.110.1130.89844.84%
InstaGen (uniform, 0.10 / 0.90, warmup)26.730.2360.84044.84%
InstaGen (uniform, 0 / 1.00, warmup)23.250.4440.73444.86%

2. Without dense warmup

Method (exact/sampled)PSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG2 (top-p 0.9), no warmup19.350.5780.72320.45%
InstaGen (uniform, 1.00 / 0.00, no warmup)13.390.5910.43920.45%
InstaGen (uniform, 0.90 / 0.10, no warmup)24.090.2220.85220.45%
InstaGen (uniform, 0.75 / 0.25, no warmup)23.560.2330.83520.45%
InstaGen (uniform, 0.50 / 0.50, no warmup)23.620.1790.86520.45%
InstaGen (uniform, 0.25 / 0.75, no warmup)19.990.3660.71920.45%
InstaGen (uniform, 0.10 / 0.90, no warmup)18.710.4660.63820.45%
InstaGen (uniform, ~0 / 1.00, no warmup)14.750.6530.48620.49%

Importance sampling ablation

We also add ablation for importance sampling below. (The overall sparsity is 32% 0.25 / 0.75 implies 8% is exact and 24% is sampled)

Method (exact/sampled)PSNR ↑SSIM ↑LPIPS ↓
InstaGen (importance, 0.0 / 1.0, warmup)26.780.9050.071
InstaGen (importance, 0.25 / 0.75, warmup)27.760.9130.060
InstaGen (importance, 0.375 / 0.625, warmup)26.310.8980.075
InstaGen (importance, 0.5 / 0.5, warmup)26.770.9070.065
InstaGen (importance, 0.625 / 0.375, warmup)27.000.9110.065
InstaGen (importance, 0.75 / 0.25, warmup)27.260.9070.064
InstaGen (importance, 1.0 / 0.0, warmup)12.100.4920.566

Larger and More Diverse Prompt Set

We added 78 VBench prompts, sampled randomly from the same eleven per-dimension lists used for the original 22 prompts (approximately seven additional prompts per list). We then evaluated Wan2.1-T2V-1.3B on all 100 prompts.

PSNR, LPIPS, and SSIM are measured against the dense video from each method's own pipeline. Density counts warmup steps as full attention. The final column reports the number of prompts, out of 100, on which InstaGen outperforms SVG-EAR (which is the strongest baseline in our experiments in the paper) for PSNR, SSIM, and LPIPS, respectively.

MethodPSNR ↑LPIPS ↓SSIM ↑Density ↓Wins vs. SVG-EAR (PSNR / SSIM / LPIPS)
SVG-EAR28.410.0720.88335.4%—
InstaGen (importance, 16+16, no warmup)29.050.0590.89832.0%61 / 65 / 73
InstaGen (importance, 16+16, warmup)30.950.0470.91752.4%78 / 87 / 86

VBench metrics on the same 100 prompts

We report all VBench metrics that can be evaluated on a generic set of prompts. However, certain VBench metrics require evaluation on their respective, predefined prompt sets. Evaluating these metrics is currently infeasible given our time and computational budget constraints.

VBench metricDenseSVG-EARInstaGen (importance, 16+16, no warmup)InstaGen (importance, 16+16, warmup)
Quality Score80.8880.4580.9280.90
Subject consistency0.97610.97260.97610.9762
Background consistency0.97400.96920.97200.9729
Temporal flickering0.98860.98840.98850.9884
Motion smoothness0.99220.99210.99220.9922
Dynamic degree0.1500.1400.1600.160
Aesthetic quality0.61770.61290.61880.6190
Imaging quality0.67660.67030.67590.6735

The improvements remain consistent on the larger prompt set and across additional VBench Metrics.

Grouped vs. Per-Query Sampling

We also experimented with implementing per-query sparse attention. However, it consistently performed worse than full attention. The key reason is that unstructured, per-query sparsity does not translate into actual computational or memory-traffic savings. The irregular sparsity patterns prevent coalesced memory accesses, while the additional overhead of managing sparsity further offsets any potential savings. This is precisely the challenge our approach addresses through group-based coalesced sampling, which produces structured sparsity that can be efficiently exploited by the hardware.

Hunyuan 1.0

To demonstrate broader applicability, we evaluate InstaGen and the comparison methods on Hunyuan 1.0.

Method (exact + sampling budget as % )PSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG-EAR34.750.0300.96540.1%
InstaGen 12+12, SVG-EAR's warmup38.290.0130.98341.2%
SVOO26.820.1010.88633.6%
InstaGen + SVOO 13+13, SVOO's warmup an d floor35.420.0240.97340.8%
Rectified17.910.3720.65320.5%
InstaGen 10+1027.960.0860.90620.0%
InstaGen 16+16, no warmup30.850.0610.92632.0%
InstaGen + SVOO 16+16, no warmup29.530.0690.91532.0%

The Hunyuan results show the same pattern across multiple sparse-attention baselines. At comparable density, InstaGen improves SVG-EAR from 34.75 to 38.29 PSNR and Rectified from 17.91 to 27.96 PSNR, while also improving LPIPS and SSIM. The no-warmup variants remain substantially stronger than the corresponding sparse baselines at 32% density.

We hope the additional results and discussion clarify your concerns. We would be happy to address any remaining questions.

Thank you to the authors for the detailed responses and additional experiments, which have addressed my concerns. The results demonstrate a better quality–efficiency trade-off than other training-free methods, including SVG2 and SVG-EAR. The additional HunyuanVideo experiments also provide evidence of applicability beyond the Wan2.1 family. Taken together, these results strengthen my assessment, and I will raise my rating to 3 (Weak Accept).

Thank you for the careful reading of our responses and for raising your rating. We are glad the additional experiments addressed your concerns. Your suggestions helped us strengthen the paper, and we will include these results in its next version.

Reviewer_SnMrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InstaGen, a training-free sparse attention method for efficient video generation. Unlike existing top-p approaches that discard low-importance attention blocks, InstaGen combines deterministic selection of important keys with importance sampling of residual keys, using inverse-probability weighting to approximate their contributions. It further introduces group-shared sampling based on query-key clustering to enable GPU-efficient execution. Experiments on Wan2.1-T2V-1.3B and 14B demonstrate improved generation fidelity and a better quality-efficiency trade-off compared to existing sparse attention methods.

Critical strengths
  1. Importance sampling provides a principled way to approximate residual attention contributions, improving accuracy over conventional top-p selection.
  2. Group-shared sampling enables structured sparsity compatible with efficient GPU execution, with minimal additional sampling overhead.
Critical weaknesses
  1. The core idea of combining deterministic selection with residual sampling has been explored in vAttention, while the clustering and permutation mechanisms are inherited from SVG2. The authors should better clarify the technical novelty beyond combining these existing techniques.

  2. InstaGen is primarily evaluated using the clustering-based sparse attention framework of SVG2. It would be valuable to investigate whether the proposed importance sampling strategy also benefits other block-wise sparse attention methods, such as MoGA[2], to demonstrate its general applicability beyond clustering-based selection.

  3. The method primarily uses fixed ratios between exact selection and residual sampling. An ablation on different budget allocations, particularly under varying attention entropy, would help justify this design choice.

[1] vAttention: Verified Sparse Attention via Sampling [2] MoGA: Mixture-of-Groups Attention for End-to-End Long Video Generation

Questions for authors

Please refer to the weaknessed part.

5 comments

Novelty

We clarify how InstaGen is different from previous approaches (especially SVG2 and vAttention) and our contributions below,

First, InstaGen is not dependent on SVG2. It can work with any existing sparse attention mechanism. The fact that InstaGen is parameterized by existing methods is not a drawback but an asset, as it allows new advances in sparse attention to be readily integrated into InstaGen. More importantly, InstaGen goes beyond simply leveraging existing methods: it addresses a key limitation of these sparse attention methods by performing residual estimation, helping achieve a better quality-efficiency tradeoff where existing methods often struggle. The gains from residual estimation are already evident in the paper. To further support this, we have added experiments combining InstaGen with SVOO and Rectified Sparse Attention.

Second, while vAttention does discuss residual estimation and a verified algorithm for it, extending the full vAttention algorithm to prefill workloads is not straightforward. Even if we borrow the idea of uniform sampling, uniform sampling independently per query cannot be efficiently accelerated on GPUs due to the lack of coalesced memory access. A straightforward extension using 2D block sampling to enable coalesced access, on the other hand, compromises quality significantly. Our contribution is in identifying the appropriate sampling structure and determining when to sample to make it work efficiently for prefill workloads. The resulting speedups and quality improvements demonstrate the value of these design choices.

More importantly, InstaGen goes beyond these ideas to improve the state of the art in sparse video generation:

  1. We extend residual estimation to importance sampling, further leveraging the underlying black-box sparse attention mechanism. This improves quality while maintaining comparable execution speed, enabling improvements in the quality-efficiency Pareto frontier.

  2. We want to emphasize that, to the best of our knowledge, we are the first to achieve fully sparse diffusion while simultaneously improving generation quality. Previous methods require dense diffusion during their initial steps. This is an important distinction in the capabilities enabled by InstaGen, which can help unlock long-video generation. (See experiments below.)

  3. For both uniform and importance sampling, we enable GPU-friendly sampling granularity which strikes a balance between efficiency and quality of estimation. This avoids uncoalesced memory access by avoiding sampling independently per query. On the other hand, it introduces correlations only along the query axis while avoiding correlations along the key axis, which is catastrophic for estimation.

4.To the best of our knowledge, we are the first to introduce stochastic residual estimation for sparse video generation, improving upon the deterministic residual estimation used in prior methods such as SVG-EAR (one centroid key per skipped cluster) and Rectified SpaAttn (pooled query-key scores).

InstaGen improves over other sparse attentions as well.

Following the suggestion, we add results for two new sparse attention mechanisms 1) SVOO and 2) Rectified SpaAttention. Regarding MoGA specifically there are challenges to integrating InstaGen with MoGA during the limited period of rebuttal primarily because the code is not yet public (their github contains only assets). However, in theory it’s a method over which InstaGen style estimation can be applied.

1. Wan2.1-T2V-1.3B

MethodPSNR ↑LPIPS ↓SSIM ↑ImgQual ↑AesQual ↑SubCons ↑BackCons ↑Density ↓
SVOO28.300.0810.9080.66010.64440.95020.958441.57%
InstaGen (importance, 8+8, warmup), SVOO’s clustering30.830.0470.9280.65090.64080.95230.960142.98%
InstaGen (importance, 16+16, no warmup), SVOO’s clustering30.940.0410.9350.65440.64490.95290.956532.00%
InstaGen (importance, 16+16, warmup), SVOO’s clustering31.530.0410.9370.65230.64260.95210.961549.32%
InstaGen (uniform, 8+8, warmup), SVOO’s clustering30.620.0470.9250.64990.64010.95200.958342.98%
InstaGen (uniform, 16+16, no warmup), SVOO’s clustering28.200.0730.9060.65010.64040.95380.961332.00%
InstaGen (uniform, 16+16, warmup), SVOO’s clustering31.330.0420.9350.65000.64080.95250.960549.32%
Rectified SpaAttn22.660.1870.8010.65030.63810.95300.958037.12%
InstaGen (importance, 8+8, warmup), Rectified’s blocks27.120.0870.8740.64920.63740.95070.957430.33%
InstaGen (importance, 16+16, no warmup), Rectified’s blocks26.990.0850.8860.64670.64250.95210.958532.08%
InstaGen (importance, 16+16, warmup), Rectified’s blocks29.410.0580.9120.64750.63780.95330.960243.33%
InstaGen (uniform, 8+8, warmup), Rectified’s blocks28.160.0720.8920.65130.63720.94900.960030.33%
InstaGen (uniform, 16+16, no warmup), Rectified’s blocks27.610.0770.8960.64970.63890.95190.960032.08%
InstaGen (uniform, 16+16, warmup), Rectified’s blocks30.020.0490.9250.65120.63750.95010.960543.33%

2. Wan2.1-T2V-14B

MethodPSNR ↑LPIPS ↓SSIM ↑ImgQual ↑AesQual ↑SubCons ↑BackCons ↑Density ↓
SVOO24.610.0970.8630.68880.66490.96630.969450.33%
InstaGen (importance, 8+8, warmup), SVOO’s clustering25.360.0830.8750.69320.66450.96640.968544.54%
InstaGen (importance, 16+16, no warmup), SVOO’s clustering23.760.1030.8450.68830.66670.97010.970932.00%
InstaGen (importance, 16+16, warmup), SVOO’s clustering25.700.0780.8830.69130.66810.96670.970050.84%
InstaGen (uniform, 8+8, warmup), SVOO’s clustering25.040.0860.8690.69170.66760.96650.968444.54%
InstaGen (uniform, 16+16, no warmup), SVOO’s clustering23.260.1060.8440.69160.66100.96860.969632.00%
InstaGen (uniform, 16+16, warmup), SVOO’s clustering25.350.0800.8780.69340.66880.96670.971050.84%

As we can see, the improvements are consistent across used underlying sparse attention and different models / settings.

Ablation on distribution over sampling and top-k budgets

We report ablations for uniform and importance sampling on Wan2.1-T2V-1.3B, evaluating uniform sampling both with and without warmup. Each split denotes how the fixed budget is divided between exact and sampled keys; for example, 0.9/0.1 allocates 90% to exact keys and 10% to sampled keys. Performance is strongest and remains stable when a larger share of the budget is allocated to exact keys. Notably, even a 0.5/0.5 split with uniform sampling outperforms the base algorithm, demonstrating robustness to the precise budget allocation.

Uniform sampling ablation

1. With dense warmup

Method (exact/sampled)PSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG2 (top-p 0.9), warmup29.610.1150.91844.79%
InstaGen (uniform, 1.00 / 0.00, warmup)27.150.1850.87544.84%
InstaGen (uniform, 0.90 / 0.10, warmup)31.830.0630.94544.84%
InstaGen (uniform, 0.75 / 0.25, warmup)31.510.0580.94344.84%
InstaGen (uniform, 0.50 / 0.50, warmup)30.710.0820.92944.84%
InstaGen (uniform, 0.25 / 0.75, warmup)29.110.1130.89844.84%
InstaGen (uniform, 0.10 / 0.90, warmup)26.730.2360.84044.84%
InstaGen (uniform, 0 / 1.00, warmup)23.250.4440.73444.86%

2. Without dense warmup

Method (exact/sampled)PSNR ↑LPIPS ↓SSIM ↑Density ↓
SVG2 (top-p 0.9), no warmup19.350.5780.72320.45%
InstaGen (uniform, 1.00 / 0.00, no warmup)13.390.5910.43920.45%
InstaGen (uniform, 0.90 / 0.10, no warmup)24.090.2220.85220.45%
InstaGen (uniform, 0.75 / 0.25, no warmup)23.560.2330.83520.45%
InstaGen (uniform, 0.50 / 0.50, no warmup)23.620.1790.86520.45%
InstaGen (uniform, 0.25 / 0.75, no warmup)19.990.3660.71920.45%
InstaGen (uniform, 0.10 / 0.90, no warmup)18.710.4660.63820.45%
InstaGen (uniform, ~0 / 1.00, no warmup)14.750.6530.48620.49%

Importance sampling ablation

We also add ablation for importance sampling below.

Method (exact/sampled)PSNR ↑SSIM ↑LPIPS ↓
InstaGen (importance, 0.0 / 1.0, warmup)26.780.9050.071
InstaGen (importance, 0.25 / 0.75, warmup)27.760.9130.060
InstaGen (importance, 0.375 / 0.625, warmup)26.310.8980.075
InstaGen (importance, 0.5 / 0.5, warmup)26.770.9070.065
InstaGen (importance, 0.625 / 0.375, warmup)27.000.9110.065
InstaGen (importance, 0.75 / 0.25, warmup)27.260.9070.064
InstaGen (importance, 1.0 / 0.0, warmup)12.100.4920.566

We hope the additional results and discussion clarify your concerns. We would be happy to address any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers (summary of changes) [1/2]

We thank all reviewers for their time and for engaging with our rebuttal. Below we summarize the main points, rebuttal and additional evidence added to the paper.

All three reviewers recognized InstaGen's empirical strength and its design. Reviewer SnMr highlighted that importance sampling gives a principled estimate of the residual attention, improving accuracy over top-p selection, and that group-shared sampling keeps the sparsity structured enough to run efficiently on GPUs, with little sampling overhead. Reviewer SJJW pointed out that InstaGen needs no training or fine-tuning, which makes it practical for existing video models, and that it stays closer to the dense model's outputs at a matched attention budget, a meaningful gain in fidelity that comes with less computation. Reviewer mQYX found the improvements over existing sparse attention methods substantial. The experiments added during the discussion strengthened this assessment further. Reviewer SnMr confirmed that the new results resolved part of their concerns. After the HunyuanVideo and 100-prompt results, Reviewer SJJW found a better quality–efficiency trade-off than other training-free methods, including SVG2 and SVG-EAR, and raised their rating to Weak Accept. After seeing InstaGen plugged into SVOO and Rectified SpaAttn, Reviewer mQYX came to view it as a general framework rather than an optimization specific to SVG2. The one remaining question, the degree of methodological novelty, we address directly in [2/2].

Main concerns and our responses.

ConcernRaised bySummary of response
Is InstaGen tied to SVG2's clustering?SnMr, mQYXNo. InstaGen works with any sparse attention mechanism. Apart from SVG2 when plugged into SVOO it gains +2.5 dB (28.30 → 30.83) and into Rectified SpaAttn, which uses no clustering, +4.5 dB (22.66 → 27.12) at lower density, on Wan2.1-1.3B; on 14B it improves SVOO at lower density.
Novelty relative to SVG2 and vAttentionSnMr, mQYXvAttention samples independently per query, for LLM decoding. On video self-attention, per-query sampling cannot use coalesced memory access and ran slower than full attention in our experiments, while 2D block sampling compromises quality significantly. InstaGen's group-shared sampling is the structure that works, and it is what lets every denoising step run sparse.
Exact vs. sampled budget splitSnMr, SJJWAblations over seven splits. With importance sampling, every split that samples is within 1.5 dB of the best and 14 dB above top-k alone; with uniform sampling, splits from 0.9/0.1 to 0.5/0.5 all beat SVG2 at the same density, with and without warm-up.
Prompt coverage and VBenchSJJW100 prompts and every VBench dimension that applies to custom prompts. With no warm-up, InstaGen beats SVG-EAR (the strongest baseline) at lower density: 29.05 vs. 28.41 dB at 32% vs. 35.4%, better on 61/65/73 of 100 prompts (PSNR/SSIM/LPIPS). Its VBench Quality Score matches dense (80.92 vs. 80.88).
Benefit of group-shared samplingSJJWWe implemented per-query sampling, and it ran slower than full attention: per-query sparsity prevents coalesced memory access, so it saves neither compute nor memory traffic. Group-shared sampling produces structured sparsity that the hardware can exploit.
Beyond Wan2.1SJJWHunyuanVideo: +3.5 dB over SVG-EAR (34.75 → 38.29) and +10 dB over Rectified SpaAttn (17.91 → 27.96) at matched density; with no warm-up, +2.7 dB over SVOO at lower density.
AuthorsOpenReview ↗

Response to all reviewers (summary of changes) [2/2]

On novelty. InstaGen takes well-known statistical tools, such as top-k selection, importance sampling and Horvitz–Thompson weighting, and combines them to estimate the residual attention that sparse selection methods skip. More importantly, InstaGen goes beyond these tools to improve the state of the art in sparse video generation:

  1. Stochastic estimation of the skipped attention in video diffusion. Prior training-free methods for video either drop the attention they skip (SVG, SVG2, SVOO) or replace it with a fixed deterministic summary (SVG-EAR, Rectified SpaAttn). InstaGen estimates it with unbiased estimates of the softmax sums, whose error keeps falling as the budget grows.
  2. The sampling structure that makes this practical. On full self-attention, both direct options fail. Sampling independently per query (vAttention's form, designed for decoding) cannot use coalesced memory access and runs slower than full attention; sampling 2D blocks of the attention map restores coalesced access but compromises quality significantly. InstaGen shares one draw across a query group but keeps individual keys.
  3. Every denoising step is sparse. Every method we compare with needs dense warm-up steps (SVG2 runs 30% of its steps dense). With none, InstaGen still outperforms SVG2, SVOO and Rectified SpaAttn run with their warm-up on Wan2.1-1.3B, and SVG-EAR on 100 prompts at lower density. A 30% warm-up caps any method's speedup at 3.3×, so this matters most for long videos: at 561 frames, InstaGen without warm-up runs 2.83× faster than dense, against 1.97× for SVG2.
  4. A plug-in improvement to block-sparse attention, now shown on SVG2, SVOO and Rectified SpaAttn, at 1.3B and 14B, and on HunyuanVideo.

Recent methods in this area, such as SVOO and SVG-EAR, also build on SVG2's framework and are judged by what they change. InstaGen changes how the skipped attention is treated, and that change is what removes the dense warm-up.

Summary of changes: new experiments.

  • InstaGen plugged into SVOO and Rectified SpaAttn on Wan2.1-T2V-1.3B, and into SVOO on Wan2.1-T2V-14B.
  • HunyuanVideo, against SVG-EAR, SVOO and Rectified SpaAttn.
  • 100 prompts with every VBench dimension that applies to custom prompts.
  • Budget-split ablations for uniform sampling (with and without warm-up) and importance sampling.
  • Per-query vs. group-shared sampling.
  • Long videos (81 to 561 frames) without warm-up.

We thank the reviewers again. Taken together, the results show that InstaGen improves the quality–efficiency trade-off of training-free sparse attention across base methods, models and prompt sets, and is to our knowledge the first to do so with every denoising step sparse ie without any warmup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.