- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RDA (Recovering Dropped Contributions from Retained Attention), a training-free sparse attention acceleration method for video generation. It addresses the quality degradation problem caused by existing sparse attention approaches that directly discard low-importance attention blocks. The core idea is to leverage retained attention outputs and approximate the contributions of dropped components via lightweight shared linear transformations, reducing computational overhead while improving the approximation accuracy of full attention outputs. Specifically, RDA consists of three main modules: (1) Shared Linear Recovery, which fits a shared linear mapping on-the-fly using full attention computation on a small number of sampled query rows, and recovers both the numerator contributions of dropped attention terms and softmax normalization terms; (2) KEXACT + VAVARe, which prioritizes retaining important and hard-to-recover attention blocks by jointly considering the attention responses of Keys and the feature variance of Values; (3) Two-Dimensional Density Scheduling, which adaptively allocates sparse computation budgets according to the sensitivity across different denoising steps and transformer layers. The authors conduct experiments on two video generation models, Wan2.1 and HunyuanVideo. The results show that at an attention sparsity of approximately 77%–78%, RDA achieves end-to-end inference speedups of 1.66× and 2.01× respectively. It outperforms the compared sparse attention methods in terms of generation quality and fidelity to the original full attention outputs. Overall, by jointly designing sparse selection, information recovery and dynamic computation budget allocation, this work offers a new technical paradigm for efficient video generation.
Critical strengths
- The research problem is clearly defined, and the proposed attention recovery idea is inspiring. Existing sparse attention methods typically directly discard unselected attention blocks, leading to information loss and degraded generation quality. Instead of simply ignoring discarded components, this paper proposes recovering their contributions from retained attention outputs. This idea carries certain novelty and offers a new optimization direction for sparse attention. Compared with directly reconstructing the full attention matrix, recovering missing contributions within attention outputs enjoys better computational feasibility.
- The method design is relatively complete, with clear logical connections among the three modules. Shared Linear Recovery fits a shared linear mapping using a small set of sampled query rows to approximately recover discarded attention contributions without retraining the model. KEXACT + VAVARe jointly considers the attention responses of Keys and the feature variance of Values, aiming to prioritize retaining important and hard-to-recover attention blocks. Two-Dimensional Density Scheduling allocates computation budgets according to the sensitivity across different denoising steps and transformer layers, further improving the trade-off between generation quality and inference efficiency.
- The experimental results are competitive and achieve a favorable balance between generation quality and inference efficiency. Evaluations are conducted on two video generation models, Wan2.1 and HunyuanVideo, covering diverse model architectures and video generation setups. At an attention sparsity of approximately 77%–78%, the method achieves end-to-end speedups of 1.66× and 2.01× respectively, while delivering superior generation fidelity compared with the sparse attention baselines. Beyond reporting overall speedup ratios, the paper also analyzes the execution efficiency of the attention kernel and additional computational overhead, demonstrating practical engineering value.
- The experimental analysis is relatively thorough, with comprehensive ablation studies and validations. It systematically evaluates the quality-speed trade-off under different sparsity levels. Held-out validation is used to verify the recovery capability of Shared Linear Recovery on unseen query rows and query groups that are not involved in fitting. Ablation experiments for block selection, recovery and density scheduling are provided, alongside end-to-end runtime breakdowns and qualitative results, which help understand the effectiveness of the proposed method.
Critical weaknesses
- The core assumption of Shared Linear Recovery lacks sufficient theoretical analysis. The central assumption of this work is that discarded attention contributions can be linearly predicted from retained attention outputs. Nevertheless, the exponential kernel feature decomposition presented in the paper only demonstrates that the two share query feature structures; it cannot directly guarantee the existence of a low-error linear mapping. Although the authors provide empirical support via held-out validation, they have not fully analyzed the conditions under which this assumption holds, nor the relationships between recovery accuracy and attention distribution, sparsity levels, or query group discrepancies. In particular, recovery errors rise noticeably under high sparsity, indicating potential limitations on the method’s applicability, yet relevant analysis remains shallow.
- Some modules yield limited standalone contributions, and ablation experiments are still insufficient. Table 7 shows that adding VAVARe upon KEXACT reduces the held-out error merely from 0.185 to 0.183, which is a marginal improvement. This is inadequate to fully justify the necessity of the value variance scoring. The main ablations in the manuscript primarily compare A0, A1 and A2. However, A1 incorporates both KEXACT + VAVARe selection and Shared Linear Recovery simultaneously, making it impossible to independently quantify each module’s contribution to the final generation quality. Moreover, the paper does not adequately verify whether value variance can reliably measure the recovery difficulty of dropped contributions.
- The design of Two-Dimensional Density Scheduling is somewhat empirical, and its generalization ability remains unclear. This module relies on offline calibration to obtain sensitivity coefficients for different denoising steps and transformer layers, which are then used to allocate sparse computation budgets. However, the calibration is performed solely on one representative generation trajectory. It is unclear whether the resulting sensitivity coefficients and density allocation strategy can generalize stably across diverse prompts, video content and motion patterns. In addition, the authors use the attention output error before recovery as the optimization objective for scheduling. Sufficient evidence is missing to demonstrate that this proxy metric has a stable correlation with the post-recovery error or final video generation quality.
- The experimental coverage remains limited, and the evaluation of video generation quality is not comprehensive enough. Experiments are conducted only on two video generation models, Wan2.1 and HunyuanVideo, lacking validation on other video generation architectures or broader tasks. Existing evaluations mainly focus on PSNR, SSIM, LPIPS and MSE with respect to full attention outputs, plus partial VBench metrics. Assessments on video temporal consistency, complex motion, and text alignment are still inadequate. Since video generation exhibits strong temporal dependencies, frame-wise image fidelity metrics alone may not fully capture the impact of sparse attention on video generation quality.
- The end-to-end efficiency gains need to be further separated from the actual benefits brought by the recovery module. Although RDA achieves promising end-to-end speedups, the performance gains come jointly from block selection, kernel optimization, Shared Linear Recovery and Density Scheduling. Existing experiments have not fully quantified the quality benefits and computational overhead introduced by Shared Linear Recovery itself under conditions with strictly matched computation budgets and identical kernel implementations. Furthermore, RDA does not deliver the fastest inference speed at all sparsity settings; in some experiments, SpargeAttn runs faster. Therefore, a clearer distinction is required between algorithm-level advantages and efficiency gains originating from low-level implementations.
- The paper does not fully follow the formatting requirements of the official ICLR template. The vertical spacing between the captions of tables and the top borders of the tables is noticeably too small, which deviates from the standard layout specified in the official ICLR template. Although the authors include an AI usage statement in Appendix I, the section is titled “AI Assistance” instead of the template-specified “AI Use Statement”, and the statement content does not fully conform to the official sample format. The authors are advised to carefully check and comply with the layout and AI disclosure requirements of the official ICLR template to ensure consistency and standardization of the submission format.
Questions for authors
- Regarding the theoretical foundation of Shared Linear Recovery The paper uses exponential kernel feature decomposition to illustrate that retained attention outputs and dropped attention contributions share query feature structures. This, however, appears insufficient to guarantee an accurate linear mapping between them. Could you further explain the conditions under which this linear recovery assumption holds? For example, does recovery accuracy depend on the rank of retained attention features, attention distribution, or similarity across different query groups? Additionally, can you analyze which attention heads or sparsity configurations yield poor recovery performance, to help clarify the applicability boundary of the method?
- Regarding the practical contribution of VAVARe Table 7 shows that adding VAVARe reduces the held-out error only from 0.185 to 0.183, which is a limited improvement. Could you provide standalone ablation results of VAVARe on final video generation quality and inference efficiency? Furthermore, are there experiments demonstrating a strong correlation between value variance and the actual recovery error of dropped contributions? This would help judge whether VAVARe is a necessary design rather than merely an empirical scoring term.
- Regarding the generalization capability of Two-Dimensional Density Scheduling The paper performs offline calibration using one representative generation trajectory and applies the resulting density scheduling strategy to other generated samples. Could you verify whether different calibration prompts lead to substantially different scheduling strategies? How much do strategies obtained from distinct calibration samples affect the final video generation quality and inference speed? Moreover, can you present comparisons among uniform density, step-only scheduling, layer-only scheduling and the full two-dimensional scheduling, to more clearly validate the necessity of the two-dimensional scheduler?
- Regarding standalone module contributions and fair comparisons A1 in the main ablation incorporates both improved block selection and Shared Linear Recovery simultaneously, making it hard to separate their individual contributions to final video generation quality. Under identical sparsity ratios and computation budgets, can you separately evaluate the performance of the original block selection + recovery, improved block selection (without recovery), and the combined version? Besides, could you clarify whether the compared methods adopt as consistent evaluation conditions as possible in terms of kernel implementations, numerical precision and statistics of extra overhead?
- Regarding the evaluation of video generation quality The paper mainly evaluates generation results using image fidelity metrics measured against the full attention baseline. Nevertheless, video quality also involves temporal consistency, motion plausibility and text alignment. Could you add more comprehensive video-level evaluations, such as metrics related to temporal consistency, motion smoothness and text alignment from VBench?
4 comments
Schedule generalization and dimensionality (W3, Q3)
1. Cross-prompt validation of the fixed schedule.
We calibrate using 50 prompts with randomly selected styles and random seeds, and validate the schedule on prompts with different video content. We further assess generation-quality stability through end-to-end evaluation on more than 100 new prompts with random seeds. Table 3 reports the mean results; the PSNR variance is 0.71, indicating stable performance.
2. Two-dimensional versus one-dimensional scheduling.
To further validate two-dimensional scheduling, we compare uniform, step-only, layer-only, and two-dimensional schedules while fixing block selection, the recovery method, and overall sparsity. Parameters are frozen before validation. Results under a fixed density budget are as follows:
| Schedule | Mean PSNR (dB) |
|---|---|
| Uniform | 21.130 |
| Step-only | 22.517 |
| Layer-only | 21.619 |
| 2D | 22.602 |
Two-dimensional scheduling achieves the highest PSNR on both validation prompts, improving the mean over uniform, step-only, and layer-only scheduling by 1.472, 0.085, and 0.983 dB, respectively. This shows that jointly considering denoising steps and network layers further improves budget allocation.
3. Consistency between the scheduling objective and post-recovery error.
We use pre-recovery attention-output error as the optimization objective because it directly measures the original information loss caused by block selection. It provides a consistent measure for comparing the effects of additional retention budgets across regions, which we combine with regional sensitivities to allocate computation. Across three prompts with different content and 24 test units, the median Spearman correlation between pre- and post-recovery errors as retained density varies is 0.801, with positive correlations in 24/24 units. These correlations, together with the end-to-end ablations, support the effectiveness of this surrogate.
Recovery benefits, overhead, and comparison fairness (W5, Q4)
1. Separating the quality benefits of selection and recovery.
The algorithmic benefit of Shared Linear Recovery is improved quality under a given sparsity budget. Its additional computational cost should be evaluated separately from the efficiency of the underlying kernel. To isolate the contributions of selection and recovery, we conduct an additional 2 × 2 ablation on Wan:
| Block selection | Without recovery | With recovery | Gain from recovery |
|---|---|---|---|
| Mean-pooling | 17.602 | 18.263 | +0.661 dB |
| KEXACT+VAWARE | 17.796 | 18.842 | +1.047 dB |
| Gain from selection | +0.194 dB | +0.579 dB | — |
Values are PSNR (dB). Recovery improves quality with either selection method. Enabling recovery further increases the gain from improved selection, demonstrating their complementary effects. This comparison matches the sparse retention budget. Sampling and fitting for recovery introduce additional computation, which we report separately from the quality gains.
2. Additional computational overhead of recovery.
The paper provides an overhead breakdown: Output recovery accounts for 14.76 s, or 2.59% of total generation time, on HunyuanVideo, with selection and recovery together accounting for 2.90%. The corresponding recovery share on Wan is 7.50%. End-to-end timing includes the actual execution costs of online selection and recovery; one-time offline calibration is reported separately.
3. Fairness of kernels, precision, and timing.
The paper separately reports isolated kernel performance and end-to-end speed–quality results, and explicitly states the precision differences: RDA uses BF16, while SpargeAttn uses INT8 QK and FP8 PV. MFU is normalized by the hardware peak at each method's precision. The appendix also reports cases where SpargeAttn is faster at the same target sparsity. RDA's advantage is the speed–quality trade-off at the reported configurations, while recovery's independent value is demonstrated by the paired quality gains above.
Video-quality evaluation (W4, Q5)
Given the limited time available during the rebuttal period, we prioritize additional evaluations of subject consistency, background consistency, and motion smoothness, with results as follows:
| Method | Subject consistency ↑ | Background consistency ↑ | Motion smoothness ↑ |
|---|---|---|---|
| SVG2 | 93.74 | 95.35 | 99.01 |
| SVG-EAR | 94.20 | 95.18 | 99.07 |
| Ours | 94.36 | 95.89 | 99.13 |
Our method outperforms the other baselines on all three metrics.
Formatting and AI disclosure (W6)
We will further check these issues and address them in the camera-ready version.
Reply to authors
Thank you for your detailed reply, most of my concerns have been addressed and the Wan ablation moves me toward acceptance. I would like to raise my rating to 3. Good luck!
Thank you for your positive feedback; we will further refine the experiments discussed above and incorporate the updated results into the camera-ready version.
Thank you for the suggestions.
1. Conditions for low-error linear recovery.
Shared query features provide a structural basis for linear recovery. Low-error recovery further requires the retained contributions to cover the main directions of query-dependent variation in the dropped contributions. Our top-p selection combines attention mass and value variation to retain important, discriminative information and support recovery of the dropped contributions. Let XR=ΦQMR and XD=ΦQMD. The optimal linear recovery residual satisfies
=\|(I-X_RX_R^\dagger)X_D\|_F.$$ Thus, recovery accuracy depends on the residual of the dropped contributions outside the column space of the retained contributions. Online fitting adapts the mapping to the current attention distribution, while regularized solving controls numerical stability. **2. Recovery across query groups.** We further validate recovery in terms of query generalization and sparsity. In the row-level (H0) and whole-group (H1) holdout experiments in Appendix A, output error on held-out queries decreases by **43.20%** and **44.40%** on average, respectively. **3. Recovery at different sparsity levels.** Using the same three prompts, six step–layer positions, and four heads, we obtain the following paired results across sparsity levels: | Local sparsity | Mean error without recovery | Mean error with recovery | Mean error reduction | |---|---:|---:|---:| | 50% | 0.001281 | 0.000777 | 39.30% | | 80% | 0.011810 | 0.007244 | 38.66% | | 94% | 0.055426 | 0.030493 | 44.98% | Less information is retained at higher sparsity, which explains the increase in absolute error. Meanwhile, **recovery reduces mean local output error at all tested sparsity levels**, demonstrating effective compensation even at approximately 94% sparsity. > Independent contributions of selection and recovery (W2, Q2) **1. Independent contribution of VAWARE.** To isolate VAWARE's contribution, we compare VAWARE disabled and enabled on HunyuanVideo while fixing KEXACT, the recovery method, prompts, and seeds. We also include a low-variance-first control. All three configurations match the number of retained blocks for each head and query block and use the same sparse-attention kernel. Mean results over test prompts are as follows: | Selection configuration | Sparsity | Mean PSNR (dB) | |---|---:|---:| | VAWARE disabled: KEXACT only | 78.190% | 22.502 | | VAWARE enabled: current method | 78.190% | **22.902** | | Low-variance-first | 78.190% | 22.311 | Enabling VAWARE improves PSNR on all three test prompts, with a mean gain of **0.399 dB**, directly validating its independent quality benefit under the same retention budget. **2. Role of value variance.** To examine the role of value variance, we retain KEXACT and reverse only the sign of the variance term, making the control prioritize low-variance blocks. Our current method achieves higher PSNR on all three prompts, with a mean gain of **0.591 dB**. This controlled comparison supports prioritizing high-variance blocks: computing blocks with greater value variation exactly helps preserve richer value information. **3. Independent and complementary benefits of selection and recovery.** To address the joint changes to selection and recovery in the main ablation, we add a 2 × 2 ablation on Wan to isolate their individual and complementary benefits: | Block selection | Without recovery | With recovery | Gain from recovery | | ------------------- | ---------------: | ------------: | -----------------: | | Mean-pooling | 17.602 | 18.263 | +0.661 dB | | KEXACT+VAWARE | 17.796 | **18.842** | **+1.047 dB** | | Gain from selection | +0.194 dB | **+0.579 dB** | — | Values are PSNR (dB). Recovery improves quality with either selection method. Enabling recovery increases the gain from improved selection from **0.194 to 0.579 dB**, demonstrating their complementary effects.