acceptodds
Under review as a conference paper at ICLR 2027

RDA: Recovering Dropped Contributions from Retained Attention for Video Generation

Abstract

The quadratic cost of self-attention is a major bottleneck in video diffusion transformers. Training-free sparse attention with online top- selection has emerged as a flexible solution, but existing methods struggle to reconcile mask adaptivity with execution efficiency. Block-wise methods enable efficient regular computation, yet their coarse partitions limit the flexibility of the resulting masks; centroid-wise methods construct more content-adaptive masks, but incur costly preprocessing and irregular computation. We propose RDA, a training-free sparse attention method that retains the efficiency of regular block-wise execution while recovering information lost by coarse block masks. RDA co-designs recovery-aware top- block selection and compensation to predict dropped contributions from already-computed retained attention outputs. We further introduce a sensitivity-driven two-dimensional scheduler that jointly assigns top- configurations across denoising steps and network layers, minimizing a calibrated attention-error objective under a target sparsity budget. On 720p HunyuanVideo and Wan2.1, RDA achieves 2.010 and 1.663 speedups at 78.3% and 77.1% sparsity, respectively, while providing the best fidelity among sparse baselines.

Primary area:
generative models
Keywords:
Diffusion Transformer (DiT), Video Generation, Sparse Attention, Training-Free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7hyWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RDA (Recovering Dropped Contributions from Retained Attention), a training-free sparse attention acceleration method for video generation. It addresses the quality degradation problem caused by existing sparse attention approaches that directly discard low-importance attention blocks. The core idea is to leverage retained attention outputs and approximate the contributions of dropped components via lightweight shared linear transformations, reducing computational overhead while improving the approximation accuracy of full attention outputs. Specifically, RDA consists of three main modules: (1) Shared Linear Recovery, which fits a shared linear mapping on-the-fly using full attention computation on a small number of sampled query rows, and recovers both the numerator contributions of dropped attention terms and softmax normalization terms; (2) KEXACT + VAVARe, which prioritizes retaining important and hard-to-recover attention blocks by jointly considering the attention responses of Keys and the feature variance of Values; (3) Two-Dimensional Density Scheduling, which adaptively allocates sparse computation budgets according to the sensitivity across different denoising steps and transformer layers. The authors conduct experiments on two video generation models, Wan2.1 and HunyuanVideo. The results show that at an attention sparsity of approximately 77%–78%, RDA achieves end-to-end inference speedups of 1.66× and 2.01× respectively. It outperforms the compared sparse attention methods in terms of generation quality and fidelity to the original full attention outputs. Overall, by jointly designing sparse selection, information recovery and dynamic computation budget allocation, this work offers a new technical paradigm for efficient video generation.

Critical strengths
  1. The research problem is clearly defined, and the proposed attention recovery idea is inspiring. Existing sparse attention methods typically directly discard unselected attention blocks, leading to information loss and degraded generation quality. Instead of simply ignoring discarded components, this paper proposes recovering their contributions from retained attention outputs. This idea carries certain novelty and offers a new optimization direction for sparse attention. Compared with directly reconstructing the full attention matrix, recovering missing contributions within attention outputs enjoys better computational feasibility.
  2. The method design is relatively complete, with clear logical connections among the three modules. Shared Linear Recovery fits a shared linear mapping using a small set of sampled query rows to approximately recover discarded attention contributions without retraining the model. KEXACT + VAVARe jointly considers the attention responses of Keys and the feature variance of Values, aiming to prioritize retaining important and hard-to-recover attention blocks. Two-Dimensional Density Scheduling allocates computation budgets according to the sensitivity across different denoising steps and transformer layers, further improving the trade-off between generation quality and inference efficiency.
  3. The experimental results are competitive and achieve a favorable balance between generation quality and inference efficiency. Evaluations are conducted on two video generation models, Wan2.1 and HunyuanVideo, covering diverse model architectures and video generation setups. At an attention sparsity of approximately 77%–78%, the method achieves end-to-end speedups of 1.66× and 2.01× respectively, while delivering superior generation fidelity compared with the sparse attention baselines. Beyond reporting overall speedup ratios, the paper also analyzes the execution efficiency of the attention kernel and additional computational overhead, demonstrating practical engineering value.
  4. The experimental analysis is relatively thorough, with comprehensive ablation studies and validations. It systematically evaluates the quality-speed trade-off under different sparsity levels. Held-out validation is used to verify the recovery capability of Shared Linear Recovery on unseen query rows and query groups that are not involved in fitting. Ablation experiments for block selection, recovery and density scheduling are provided, alongside end-to-end runtime breakdowns and qualitative results, which help understand the effectiveness of the proposed method.
Critical weaknesses
  1. The core assumption of Shared Linear Recovery lacks sufficient theoretical analysis. The central assumption of this work is that discarded attention contributions can be linearly predicted from retained attention outputs. Nevertheless, the exponential kernel feature decomposition presented in the paper only demonstrates that the two share query feature structures; it cannot directly guarantee the existence of a low-error linear mapping. Although the authors provide empirical support via held-out validation, they have not fully analyzed the conditions under which this assumption holds, nor the relationships between recovery accuracy and attention distribution, sparsity levels, or query group discrepancies. In particular, recovery errors rise noticeably under high sparsity, indicating potential limitations on the method’s applicability, yet relevant analysis remains shallow.
  2. Some modules yield limited standalone contributions, and ablation experiments are still insufficient. Table 7 shows that adding VAVARe upon KEXACT reduces the held-out error merely from 0.185 to 0.183, which is a marginal improvement. This is inadequate to fully justify the necessity of the value variance scoring. The main ablations in the manuscript primarily compare A0, A1 and A2. However, A1 incorporates both KEXACT + VAVARe selection and Shared Linear Recovery simultaneously, making it impossible to independently quantify each module’s contribution to the final generation quality. Moreover, the paper does not adequately verify whether value variance can reliably measure the recovery difficulty of dropped contributions.
  3. The design of Two-Dimensional Density Scheduling is somewhat empirical, and its generalization ability remains unclear. This module relies on offline calibration to obtain sensitivity coefficients for different denoising steps and transformer layers, which are then used to allocate sparse computation budgets. However, the calibration is performed solely on one representative generation trajectory. It is unclear whether the resulting sensitivity coefficients and density allocation strategy can generalize stably across diverse prompts, video content and motion patterns. In addition, the authors use the attention output error before recovery as the optimization objective for scheduling. Sufficient evidence is missing to demonstrate that this proxy metric has a stable correlation with the post-recovery error or final video generation quality.
  4. The experimental coverage remains limited, and the evaluation of video generation quality is not comprehensive enough. Experiments are conducted only on two video generation models, Wan2.1 and HunyuanVideo, lacking validation on other video generation architectures or broader tasks. Existing evaluations mainly focus on PSNR, SSIM, LPIPS and MSE with respect to full attention outputs, plus partial VBench metrics. Assessments on video temporal consistency, complex motion, and text alignment are still inadequate. Since video generation exhibits strong temporal dependencies, frame-wise image fidelity metrics alone may not fully capture the impact of sparse attention on video generation quality.
  5. The end-to-end efficiency gains need to be further separated from the actual benefits brought by the recovery module. Although RDA achieves promising end-to-end speedups, the performance gains come jointly from block selection, kernel optimization, Shared Linear Recovery and Density Scheduling. Existing experiments have not fully quantified the quality benefits and computational overhead introduced by Shared Linear Recovery itself under conditions with strictly matched computation budgets and identical kernel implementations. Furthermore, RDA does not deliver the fastest inference speed at all sparsity settings; in some experiments, SpargeAttn runs faster. Therefore, a clearer distinction is required between algorithm-level advantages and efficiency gains originating from low-level implementations.
  6. The paper does not fully follow the formatting requirements of the official ICLR template. The vertical spacing between the captions of tables and the top borders of the tables is noticeably too small, which deviates from the standard layout specified in the official ICLR template. Although the authors include an AI usage statement in Appendix I, the section is titled “AI Assistance” instead of the template-specified “AI Use Statement”, and the statement content does not fully conform to the official sample format. The authors are advised to carefully check and comply with the layout and AI disclosure requirements of the official ICLR template to ensure consistency and standardization of the submission format.
Questions for authors
  1. Regarding the theoretical foundation of Shared Linear Recovery The paper uses exponential kernel feature decomposition to illustrate that retained attention outputs and dropped attention contributions share query feature structures. This, however, appears insufficient to guarantee an accurate linear mapping between them. Could you further explain the conditions under which this linear recovery assumption holds? For example, does recovery accuracy depend on the rank of retained attention features, attention distribution, or similarity across different query groups? Additionally, can you analyze which attention heads or sparsity configurations yield poor recovery performance, to help clarify the applicability boundary of the method?
  2. Regarding the practical contribution of VAVARe Table 7 shows that adding VAVARe reduces the held-out error only from 0.185 to 0.183, which is a limited improvement. Could you provide standalone ablation results of VAVARe on final video generation quality and inference efficiency? Furthermore, are there experiments demonstrating a strong correlation between value variance and the actual recovery error of dropped contributions? This would help judge whether VAVARe is a necessary design rather than merely an empirical scoring term.
  3. Regarding the generalization capability of Two-Dimensional Density Scheduling The paper performs offline calibration using one representative generation trajectory and applies the resulting density scheduling strategy to other generated samples. Could you verify whether different calibration prompts lead to substantially different scheduling strategies? How much do strategies obtained from distinct calibration samples affect the final video generation quality and inference speed? Moreover, can you present comparisons among uniform density, step-only scheduling, layer-only scheduling and the full two-dimensional scheduling, to more clearly validate the necessity of the two-dimensional scheduler?
  4. Regarding standalone module contributions and fair comparisons A1 in the main ablation incorporates both improved block selection and Shared Linear Recovery simultaneously, making it hard to separate their individual contributions to final video generation quality. Under identical sparsity ratios and computation budgets, can you separately evaluate the performance of the original block selection + recovery, improved block selection (without recovery), and the combined version? Besides, could you clarify whether the compared methods adopt as consistent evaluation conditions as possible in terms of kernel implementations, numerical precision and statistics of extra overhead?
  5. Regarding the evaluation of video generation quality The paper mainly evaluates generation results using image fidelity metrics measured against the full attention baseline. Nevertheless, video quality also involves temporal consistency, motion plausibility and text alignment. Could you add more comprehensive video-level evaluations, such as metrics related to temporal consistency, motion smoothness and text alignment from VBench?
4 comments

Thank you for the suggestions.

Conditions for linear recovery and sparsity analysis (W1, Q1)

1. Conditions for low-error linear recovery.

Shared query features provide a structural basis for linear recovery. Low-error recovery further requires the retained contributions to cover the main directions of query-dependent variation in the dropped contributions. Our top-p selection combines attention mass and value variation to retain important, discriminative information and support recovery of the dropped contributions. Let XR=ΦQMRX_R=\Phi_QM_R and XD=ΦQMDX_D=\Phi_QM_D. The optimal linear recovery residual satisfies

=\|(I-X_RX_R^\dagger)X_D\|_F.$$ Thus, recovery accuracy depends on the residual of the dropped contributions outside the column space of the retained contributions. Online fitting adapts the mapping to the current attention distribution, while regularized solving controls numerical stability. **2. Recovery across query groups.** We further validate recovery in terms of query generalization and sparsity. In the row-level (H0) and whole-group (H1) holdout experiments in Appendix A, output error on held-out queries decreases by **43.20%** and **44.40%** on average, respectively. **3. Recovery at different sparsity levels.** Using the same three prompts, six step–layer positions, and four heads, we obtain the following paired results across sparsity levels: | Local sparsity | Mean error without recovery | Mean error with recovery | Mean error reduction | |---|---:|---:|---:| | 50% | 0.001281 | 0.000777 | 39.30% | | 80% | 0.011810 | 0.007244 | 38.66% | | 94% | 0.055426 | 0.030493 | 44.98% | Less information is retained at higher sparsity, which explains the increase in absolute error. Meanwhile, **recovery reduces mean local output error at all tested sparsity levels**, demonstrating effective compensation even at approximately 94% sparsity. > Independent contributions of selection and recovery (W2, Q2) **1. Independent contribution of VAWARE.** To isolate VAWARE's contribution, we compare VAWARE disabled and enabled on HunyuanVideo while fixing KEXACT, the recovery method, prompts, and seeds. We also include a low-variance-first control. All three configurations match the number of retained blocks for each head and query block and use the same sparse-attention kernel. Mean results over test prompts are as follows: | Selection configuration | Sparsity | Mean PSNR (dB) | |---|---:|---:| | VAWARE disabled: KEXACT only | 78.190% | 22.502 | | VAWARE enabled: current method | 78.190% | **22.902** | | Low-variance-first | 78.190% | 22.311 | Enabling VAWARE improves PSNR on all three test prompts, with a mean gain of **0.399 dB**, directly validating its independent quality benefit under the same retention budget. **2. Role of value variance.** To examine the role of value variance, we retain KEXACT and reverse only the sign of the variance term, making the control prioritize low-variance blocks. Our current method achieves higher PSNR on all three prompts, with a mean gain of **0.591 dB**. This controlled comparison supports prioritizing high-variance blocks: computing blocks with greater value variation exactly helps preserve richer value information. **3. Independent and complementary benefits of selection and recovery.** To address the joint changes to selection and recovery in the main ablation, we add a 2 × 2 ablation on Wan to isolate their individual and complementary benefits: | Block selection | Without recovery | With recovery | Gain from recovery | | ------------------- | ---------------: | ------------: | -----------------: | | Mean-pooling | 17.602 | 18.263 | +0.661 dB | | KEXACT+VAWARE | 17.796 | **18.842** | **+1.047 dB** | | Gain from selection | +0.194 dB | **+0.579 dB** | — | Values are PSNR (dB). Recovery improves quality with either selection method. Enabling recovery increases the gain from improved selection from **0.194 to 0.579 dB**, demonstrating their complementary effects.

Schedule generalization and dimensionality (W3, Q3)

1. Cross-prompt validation of the fixed schedule.

We calibrate using 50 prompts with randomly selected styles and random seeds, and validate the schedule on prompts with different video content. We further assess generation-quality stability through end-to-end evaluation on more than 100 new prompts with random seeds. Table 3 reports the mean results; the PSNR variance is 0.71, indicating stable performance.

2. Two-dimensional versus one-dimensional scheduling.

To further validate two-dimensional scheduling, we compare uniform, step-only, layer-only, and two-dimensional schedules while fixing block selection, the recovery method, and overall sparsity. Parameters are frozen before validation. Results under a fixed density budget are as follows:

ScheduleMean PSNR (dB)
Uniform21.130
Step-only22.517
Layer-only21.619
2D22.602

Two-dimensional scheduling achieves the highest PSNR on both validation prompts, improving the mean over uniform, step-only, and layer-only scheduling by 1.472, 0.085, and 0.983 dB, respectively. This shows that jointly considering denoising steps and network layers further improves budget allocation.

3. Consistency between the scheduling objective and post-recovery error.

We use pre-recovery attention-output error as the optimization objective because it directly measures the original information loss caused by block selection. It provides a consistent measure for comparing the effects of additional retention budgets across regions, which we combine with regional sensitivities to allocate computation. Across three prompts with different content and 24 test units, the median Spearman correlation between pre- and post-recovery errors as retained density varies is 0.801, with positive correlations in 24/24 units. These correlations, together with the end-to-end ablations, support the effectiveness of this surrogate.

Recovery benefits, overhead, and comparison fairness (W5, Q4)

1. Separating the quality benefits of selection and recovery.

The algorithmic benefit of Shared Linear Recovery is improved quality under a given sparsity budget. Its additional computational cost should be evaluated separately from the efficiency of the underlying kernel. To isolate the contributions of selection and recovery, we conduct an additional 2 × 2 ablation on Wan:

Block selectionWithout recoveryWith recoveryGain from recovery
Mean-pooling17.60218.263+0.661 dB
KEXACT+VAWARE17.79618.842+1.047 dB
Gain from selection+0.194 dB+0.579 dB—

Values are PSNR (dB). Recovery improves quality with either selection method. Enabling recovery further increases the gain from improved selection, demonstrating their complementary effects. This comparison matches the sparse retention budget. Sampling and fitting for recovery introduce additional computation, which we report separately from the quality gains.

2. Additional computational overhead of recovery.

The paper provides an overhead breakdown: Output recovery accounts for 14.76 s, or 2.59% of total generation time, on HunyuanVideo, with selection and recovery together accounting for 2.90%. The corresponding recovery share on Wan is 7.50%. End-to-end timing includes the actual execution costs of online selection and recovery; one-time offline calibration is reported separately.

3. Fairness of kernels, precision, and timing.

The paper separately reports isolated kernel performance and end-to-end speed–quality results, and explicitly states the precision differences: RDA uses BF16, while SpargeAttn uses INT8 QK and FP8 PV. MFU is normalized by the hardware peak at each method's precision. The appendix also reports cases where SpargeAttn is faster at the same target sparsity. RDA's advantage is the speed–quality trade-off at the reported configurations, while recovery's independent value is demonstrated by the paired quality gains above.

Video-quality evaluation (W4, Q5)

Given the limited time available during the rebuttal period, we prioritize additional evaluations of subject consistency, background consistency, and motion smoothness, with results as follows:

MethodSubject consistency ↑Background consistency ↑Motion smoothness ↑
SVG293.7495.3599.01
SVG-EAR94.2095.1899.07
Ours94.3695.8999.13

Our method outperforms the other baselines on all three metrics.

Formatting and AI disclosure (W6)

We will further check these issues and address them in the camera-ready version.

Reviewer_7hyWOpenReview ↗

Reply to authors

Thank you for your detailed reply, most of my concerns have been addressed and the Wan ablation moves me toward acceptance. I would like to raise my rating to 3. Good luck!

Thank you for your positive feedback; we will further refine the experiments discussed above and incorporate the updated results into the camera-ready version.

Reviewer_jXSNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RDA, a training-free sparse attention method for video Diffusion Transformers. RDA jointly designs recovery-oriented block selection, compensation for dropped attention contributions, and two-dimensional sparsity-budget scheduling. Specifically, RDA uses KEXACT and VAWARE to select important blocks, and fits a shared linear mapping for each attention head using sampled full-attention rows to predict the dropped contributions from the retained outputs. It further employs a two-dimensional step-layer scheduler to allocate the sparsity budget. Experiments show that RDA achieves 2.010× and 1.663× end-to-end speedups on HunyuanVideo and Wan2.1 at approximately 78% and 77% sparsity, respectively, while maintaining competitive output fidelity among the evaluated sparse-attention baselines.

Critical strengths
  • The idea of predicting dropped attention contributions from retained attention outputs is novel and avoids directly relying on dropped-side clustering or centroid-based compensation. The method jointly designs regular block execution, recovery-oriented block selection, and step-layer scheduling, achieving a favorable balance between GPU efficiency and generation quality.

  • The system implementation is relatively comprehensive, including customized GPU kernels, numerical safeguards, and an end-to-end latency breakdown. RDA achieves competitive speed-quality trade-offs on two representative 720p video generation models, and the ablation studies generally support the effectiveness of its main components.

Critical weaknesses
  • Motivated by the observation that the retained-side and dropped-side outputs share the same query features, RDA predicts the dropped contribution using the linear mapping XD≈XRT~X_D\approx X_R\widetilde T. However, the shared feature structure in Equation (3) alone does not establish whether a stable and generalizable shared linear mapping consistently exists under different retained/dropped partitions. The current validation lacks a systematic generalization analysis across prompts, random seeds, and input distributions. Moreover, the recovery safeguard evaluates the fitted mapping on the same sampled rows used to estimate T~\widetilde T, which may fail to detect overfitting or recovery failures on unsampled query rows.

  • The two-dimensional scheduler requires model-specific offline calibration, but the independence and representativeness of the calibration data, as well as the stability of the resulting schedule across different settings, remain unclear. In addition, the scheduler uses the pre-compensation local error LZ(d)L_Z(d) as a surrogate for post-compensation output quality, but the consistency between these two quantities has not been sufficiently validated.

  • The generation-quality evaluation and baseline comparisons are not sufficiently comprehensive. The paper reports only the Imaging Quality and Dynamic Degree dimensions of VBench, while omitting important aspects such as temporal consistency and text-video alignment. Furthermore, the experiments do not compare against concurrent training-free methods with dropped-contribution approximation, such as PISA and PASA.

Questions for authors
  1. Under what conditions does the shared feature representation ΦQ\Phi_Q in Equation (3) support a stable mapping XD≈XRT~X_D\approx X_R\widetilde T? Is this linear recovery relationship stable across different prompts, random seeds, video content, and input distributions? Could the authors report recovery-error distributions across denoising steps, layers, attention heads, and prompts, as well as the proportion of heads or attention calls for which recovery is accepted?

  2. Why does the recovery acceptance test use the same sampled rows that are used to fit T~\widetilde T? Could the authors instead evaluate recovery using an independent set of held-out rows and report the proportion of heads or attention calls for which recovery is rejected or falls back to the retained-only output?

  3. How many prompts and random seeds are used for offline calibration? Is the calibration prompt set completely disjoint from the 50 prompts used for final evaluation? How much do the calibrated regional sensitivities and top-p thresholds vary across different prompt sets and random seeds? As the number of calibration prompts increases, how many samples are required before the resulting schedule and quality-sparsity curves become stable?

  4. Why does the scheduler use the pre-compensation error LZ(d)L_Z(d) rather than the post-compensation error? How strongly are these two quantities correlated with the final video-generation error?

  5. Could the authors report a more comprehensive set of VBench metrics? Currently, only two VBench dimensions are presented.

  6. Could the authors provide comparisons against training-free sparse attention methods that similarly use regular block partitioning and approximate dropped attention contributions, such as PISA and PASA, under matched-sparsity, matched-latency, or matched-fidelity settings?

1 comment

Thank you for the suggestions.

Generalization and recovery safeguards (W1, Q1–Q2)

1. Generalization of the shared mapping and recovery by group (Q1).

As shown in Appendix A, we evaluate the shared mapping on held-out queries using row-level holdout (H0) and whole-group holdout (H1). H1 completely excludes the test query groups from fitting and achieves recovery performance close to H0, supporting the mapping's applicability across query groups and their different retained/dropped partitions. We also conduct VBench evaluation on 251 prompts, with a stability evaluation on 60 prompts selected by stratified random sampling, each with 3 different generation seeds.

Holdout protocolMedian captureMean output error reduction
H0: row-level holdout73.53%43.20%
H1: whole-group holdout73.82%44.40%

We further analyze the H1 whole-group holdout results in Appendix A by group. On query groups completely excluded from fitting, all 24 evaluated units pass the safeguard and achieve lower output error, with a mean reduction of 44.40%, supporting the shared mapping's generalization within the evaluated scope.

Grouping dimensionGroupsMean output error reduction
Step20 / 3543.11% / 45.69%
Head0 / 6 / 12 / 1855.87% / 87.16% / 25.52% / 9.06%
Promptp0 / p7 / p2950.57% / 39.48% / 43.16%
OverallAll 24 units44.40%

2. Validation independent of the safeguard evaluated on the fitting rows (Q2).

To address the concern about the safeguard evaluated on the fitting rows, we also directly evaluate recovery on 111,375 unsampled queries per head. The mean output error decreases by 60.66%, showing that recovery benefits extend beyond the fitting samples and providing validation independent of the safeguard evaluated on the fitting rows.

Output MSE on unsampled queriesWithout recoveryWith recoveryReduction
Mean0.0017180.00067660.66%

Schedule calibration and scheduling objective (W2, Q3–Q4)

1. Offline calibration and cross-prompt validation (Q3).

We calibrate using 50 prompts with randomly selected styles and random seeds, and validate the schedule on prompts with different video content. We further assess generation-quality stability through end-to-end evaluation on more than 100 new prompts with random seeds. Table 3 reports the mean results; the PSNR variance is 0.71, indicating stable performance.

2. Rationale for using pre-recovery error as the scheduling objective (Q4).

We use pre-recovery attention-output error as the optimization objective because it directly measures the original information loss caused by block selection. It provides a consistent measure for comparing the effects of additional retention budgets across regions, which we combine with regional sensitivities to allocate computation. Across three prompts with different content and 24 test units, the median Spearman correlation between pre- and post-recovery errors as retained density varies is 0.801, with positive correlations in 24/24 units. These correlations, together with the end-to-end ablations, support the effectiveness of this surrogate.

Video-quality evaluation and additional comparisons (W3, Q5–Q6)

1. Additional video-quality evaluation (Q5).

Given the limited time available during the rebuttal period, we prioritize additional evaluations of subject consistency, background consistency, and motion smoothness, with results as follows:

MethodSubject consistency ↑Background consistency ↑Motion smoothness ↑
SVG293.7495.3599.01
SVG-EAR94.2095.1899.07
Ours94.3695.8999.13

Our method outperforms the baselines on all three metrics.

2. Comparisons with PISA and PASA (Q6).

The official PISA implementations for Wan and HunyuanVideo are currently still listed as pending release. We have not found the corresponding PASA paper and would appreciate further information from the reviewer.

Reviewer_JKoSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RDA is a training-free sparse-attention method for video diffusion transformers that predicts dropped contributions from retained attention outputs. Within each attention call, it computes full attention on sampled query rows and fits a shared per-head linear readout to recover omitted value sums and normalization masses. It combines this recovery mechanism with regular block selection based on individual-key responses and value heterogeneity, plus an offline schedule that allocates attention density across denoising steps and network layers. Evaluation on HunyuanVideo and Wan reports improved fidelity and end-to-end speed over the evaluated sparse baselines. Supporting experiments include whole-query-group holdouts, controlled comparisons of selection and recovery methods, sparsity sweeps, and kernel profiling.

Critical strengths
  • The recovery design offers a practical way to compensate for dropped attention while retaining regular block execution. Predicting omitted contributions from retained outputs provides a useful alternative to centroid-based compensation and avoids the clustering and reordering required by the compared centroid-based methods.

  • The implementation gives the recovery approach practical systems relevance. Regular blocks preserve efficient GPU execution while a predictor fitted on sampled queries estimates the missing contributions. The main kernel measurements demonstrate effective execution across sparsity levels. End-to-end results support the feasibility of combining online prediction with acceleration.

Critical weaknesses
  • The contribution of the recovery mechanism is not sufficiently isolated. The main ablation changes block selection and recovery together, leaving recovery’s independent benefit unclear. It also lacks a controlled comparison with direct output correction, thus the added value of predicting unnormalized value sums and normalization masses remains unestablished.

  • The shared-feature motivation does not sufficiently explain when dropped contributions are predictable. The kernel factorization establishes shared query dependence, but this alone does not imply an accurate linear map between retained and dropped outputs. The reported holdout results support feasibility without identifying the conditions that make recovery succeed. The error variation in Figure 2 suggests further analysis when recovery is less effective.

  • The main experimental protocol does not sufficiently establish comparison fairness and robustness. Baseline and sparsity configurations are insufficiently specified.

Questions for authors
  • With selection and scheduling fixed, what is recovery’s independent benefit, and does augmented-output recovery outperform direct correction of normalized outputs?

  • What properties of the retained outputs and selected supports determine whether a shared linear readout can accurately recover the dropped contribution?

  • Can you clarify the baseline configurations? Do the reported advantages remain consistent?

3 comments

Thank you for the suggestions.

Independent recovery benefits and recovery formulation (W1, Q1)

1. Independent benefit of recovery with block selection fixed.

Block selection and recovery are co-designed: selection determines both the retained-side predictive features and the dropped-side recovery targets. To further isolate recovery's contribution and demonstrate their interaction, we conducted an additional 2 × 2 ablation:

Block selectionWithout recoveryWith recoveryGain from recovery
Mean-pooling17.60218.263+0.661 dB
KEXACT+VAWARE17.79618.842+1.047 dB
Gain from selection+0.194 dB+0.579 dB—

Values are PSNR (dB). Recovery improves quality with either selection method, with a larger gain under our proposed selection. Enabling recovery also increases the gain from selection from 0.194 to 0.579 dB, demonstrating their complementary effects.

2. Augmented recovery versus direct correction of normalized outputs.

We use the additive structure of attention:

O=NR+NDZR+ZD.O=\frac{N_R+N_D}{Z_R+Z_D}.

Dropping attention terms removes value contributions and changes the normalization denominator, with query-dependent effects. Our augmented recovery predicts both components and combines them using the original attention formula. Compared with directly applying a linear correction to normalized outputs, it explicitly handles query-dependent normalization adjustments and offers greater expressiveness.

We compare the two recovery formulations on HunyuanVideo with identical inputs, block selection, sampling budgets, and solvers. The table below reports relative output errors for two representative cases:

CaseDirect correctionOurs recoveryError reduction
Case 10.0141050.01188915.71%
Case 20.0381510.03347112.27%

Conditions for shared recovery and head-level results (W2, Q2)

1. Conditions for shared linear recovery.

Accurate recovery depends on whether linear combinations of retained outputs can represent the main query-dependent variations in dropped contributions, and whether the sampled rows sufficiently cover these variations. Shared query features provide the structural motivation. KEXACT prioritizes strong attention responses, while VAWARE prioritizes blocks with greater value variation. Together, they aim to compute important, harder-to-approximate information exactly and provide a basis for recovering the remaining contributions. We sample across query groups and refit the mapping within each attention call to adapt it to the current input.

In the appendix, whole-group and row-level holdouts achieve median capture rates of 73.82% and 73.53%, respectively, supporting the applicability of the shared mapping to query groups excluded from fitting.

2. Recovery across attention heads.

Figure 2 shows residual errors after recovery. We additionally provide paired results before and after recovery. The following table compares different heads on held-out queries:

HeadRelative output error before recoveryRelative output error after recoveryError reduction
00.0114480.00614746.31%
60.1520430.02633682.68%
120.0108320.00579346.52%
180.0002120.00019010.25%

All these heads improve. Head 6 has a larger initial error, which recovery reduces by 82.68%. Head 18 starts with an error of only 0.000212, which further decreases to 0.000190. Considering both the errors before and after recovery and their relative reductions gives a more complete picture of the benefits across heads.

Baseline configurations and evaluation conditions (W3, Q3)

1. Baseline configurations.

The configurations used in the main Wan2.1 and HunyuanVideo experiments are as follows:

ModelMethodTop-k / Top-pK-means clusters (Q / K)Minimum retention ratio
Wan2.1SpargeAttnTop-k = 0.5——
Wan2.1SVG2Top-p = 0.75400 / 20000.02
Wan2.1SVG-EARTop-p = 0.30400 / 10000.058
HunyuanVideoSpargeAttnTop-k = 0.5——
HunyuanVideoSVG2Top-p = 0.75400 / 10000.02
HunyuanVideoSVG-EARTop-p = 0.85400 / 10000.02

Both SVG2 and SVG-EAR use 50 iterations for K-means initialization and 2 iterations for subsequent updates.

2. Consistent evaluation conditions.

All methods use 720p resolution, 50 sampling steps, and the same prompts and random seeds. Wan2.1 and HunyuanVideo generate 81 and 129 frames, respectively. Quality metrics are computed against FA3 full-attention outputs from each method's own pipeline using the same prompt and seed.

The authors have addressed most of my concerns, especially with the additional detailed analysis. I will raise my score.

Thank you for your positive feedback and we will further refine the experiments discussed above and incorporate the updated results into the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.