acceptodds
Under review as a conference paper at ICLR 2027

Video DeltaNet: A Video-Native Hybrid Attention for Efficient Video Generation

19 views

Abstract

Video diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention a major computational bottleneck. Linear attention offers an appealing alternative and has been widely adopted in recent large language models, but directly applying it to video models often fails to preserve the fine-grained interactions required for high-quality generation. We present Video DeltaNet (VDN), which combines local Softmax attention with bidirectional linear memory for long-range video context. Its linear branch introduces Video Delta Attention (VDA), which jointly resolves correlated within-frame writes and yields a non-expansive inherited-state transition without additional frame-size key scaling. Separate output projections and learnable gates calibrate the two branches, while a staged teacher-alignment recipe progressively introduces the new pathway into pretrained models. We instantiate VDN on MiniMax H3, applying the hybrid to video-to-video interactions while retaining Softmax for interactions involving text or audio. Applying VDN directly yields 2.6× and 3.2× backbone speedups on B200 and H200, respectively. With 8-NFE distillation and optimized inference, VDN-H3 completes DiT denoising for a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs, corresponding to a 14.5× speedup over the 50-NFE Dense H3 baseline on the same GPU count.

Primary area:
generative models
Keywords:
Video generation, linear attention, efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nBo8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper the authors presents Video DeltaNet, a hybrid attention method for accelerating video diffusion models. The proposed method uses Softmax attention for nearby frames and global boundary frames, while distant video context is represented using bidirectional linear memory. The proposed Video Delta Attention updates the memory jointly for all spatial tokens in a frame, accounting for correlations between their writes. The paper also introduces a staged procedure for adapting the pretrained MiniMax H3 model, followed by a step distillation and several kernel optimizations. The experiments show notable practical speedups on H200 and B200 GPUs, while the reported automatic quality metrics remain generally close to those of the original Dense H3 model on most of the metrics.

Critical strengths
  1. The frame-wise delta update is well motivated: The method jointly handles all spatial tokens of a frame and accounts for correlations between their writes. This is more natural for video than applying token-wise delta updates using an arbitrary spatial order and is backed up by observed advantages in empirical evaluation.

  2. The paper reports strong practical speedups: The authors provide full backbone latency on both H200 and B200, scaling results for different video lengths, and detailed kernel ablations. The reported speedups are therefore supported by actual system measurements, rather than only theoretical complexity.

  3. The hybrid design is practical: Local interactions remain under Softmax attention, while distant context is handled by linear memory. The first and last frames are also kept as global anchors. This seems like a reasonable way to reduce attention cost while preserving important local and boundary information.

  4. The paper presents a complete conversion and deployment pipeline: It covers architecture adaptation, step distillation, Triton kernel optimization, and multi-GPU execution. This makes the work more useful than a method evaluated only at the operator level.

Critical weaknesses
  1. The visual quality evaluation is not sufficient: While automatic benchmark metrics surely bring values to the empirical evaluations, their limitations in matching human perceptual preference, which I believe is the ultimate goal for the video generation, is known to the community. The paper offers no human evaluation studies. This is important because preserving the visual and temporal quality of Dense H3 is one of the main claims.

  2. The baseline comparison is narrow: At 50 NFE, the main efficient baseline is Radial Attention, which is a training-free efficient attention method, while VDN uses staged training. The experimental comparison would be stronger with a few training-based efficient attention baselines, e.g. VSA, SLA, SLA2, etc.. The paper also does not discuss some of the closely related hybrid recurrent methods e.g. ARL2 and ReHyAt, both of which combine local Softmax attention with recurrent linear memory for longer range video context. A discussion would help clarify the novelty, distinction and the extra value of the proposed hybrid design over the existing works.

  3. Some important design choices are not ablated: Most importantly, there is no forward-only variant showing the contribution of the reverse linear scan. This also matters for deployment because the reverse scan requires access to future latent chunks and prevents causal chunk-wise generation. Furthermore, the proposed three-stage adaptation pipeline is also not ablated with respect to the final quality. The parameter displacement analysis does not show if A1, A2, and the LoRA-based stage B are all needed; another aspect that I didn't find to be directly motivated intuitively or empirically is the use of LoRA for the tuning in stage B.

Questions for authors
  1. Subjective evaluation: Could the authors provide a method-blind human preference study between VDN-H3 and Dense H3? This evidence can directly affect my assessment of the quality-preservation claim, even if done at a moderate scale.

  2. Instruction following before and after step distillation: At 50 NFE, VDN-H3 is lower than Dense H3 and Radial Attention on FIRM instruction following, while the 8 NFE model appears to recover strongly on this metric. I am curious whether this improvement comes from DMD distillation, initialization from the H3-Turbo LoRA, or another part of the training pipeline. This could clarify whether the 50 NFE result reflects the attention architecture or its adaptation.

  3. Teacher selection for step distillation: The 8 NFE student is distilled from the already converted 50 NFE VDN H3 model rather than directly from Dense H3. I understand that this separates architecture conversion from step reduction. However, I wonder whether using Dense H3 as the teacher could help recover some of the quality lost during architecture conversion. Was this alternative tested or considered?

4 comments

We are grateful to Reviewer nBo8 for the thoughtful assessment. Below, we respond to each weakness and question in turn; Question 1 is answered together with Weakness 1.

Weakness 1 & Question 1: Human evaluation of visual quality

Weakness 1. The visual quality evaluation is not sufficient: While automatic benchmark metrics surely bring values to the empirical evaluations, their limitations in matching human perceptual preference, which I believe is the ultimate goal for the video generation, is known to the community. The paper offers no human evaluation studies. This is important because preserving the visual and temporal quality of Dense H3 is one of the main claims.

Question 1. Subjective evaluation: Could the authors provide a method-blind human preference study between VDN-H3 and Dense H3? This evidence can directly affect my assessment of the quality-preservation claim, even if done at a moderate scale.

We have added a method-blind, pairwise human preference study with 10 evaluators and 103 video pairs per model comparison. Model names were hidden; evaluators selected either video or a tie to record overall preference.

JudgmentsDense H3 50 NFETieVDN-H3 50 NFE
All judgments22.3%59.2%18.4%
JudgmentsVDN-H3 8 NFETieFastH3 v2 8 NFE
All judgments62.1%12.6%25.2%

At 50 NFE, 59.2% of judgments are ties, suggesting that VDN-H3 largely preserves the perceptual quality of Dense H3. Together with the matched-resource speedups reported in Table 2, this supports the quality–efficiency trade-off of our attention design. At 8 NFE, evaluators favor VDN-H3 in 62.1% of judgments and FastH3 v2 in 25.2%, with 12.6% ties. This clear preference in the separate few-step comparison indicates that the advantages of the VDN-H3 system are perceptible to human viewers, supporting its practical value for efficient video generation.

Weakness 2: Baselines and related work

The baseline comparison is narrow: At 50 NFE, the main efficient baseline is Radial Attention, which is a training-free efficient attention method, while VDN uses staged training. The experimental comparison would be stronger with a few training-based efficient attention baselines, e.g. VSA, SLA, SLA2, etc.. The paper also does not discuss some of the closely related hybrid recurrent methods e.g. ARL2 and ReHyAt, both of which combine local Softmax attention with recurrent linear memory for longer range video context. A discussion would help clarify the novelty, distinction and the extra value of the proposed hybrid design over the existing works.

The submission already includes a training-based VSA baseline: FastH3 v2 uses VSA-H3 with 80% sparsity and DMD2 training. At the same 8 NFE, VDN-H3 achieves higher VQA_A, VQA_T, Q-Align, and FAST-VQA scores, as shown below. Its one-B200 denoising latency is also lower: 49.3 s versus 60.2 s for FastH3 v2.

MethodVQA_AVQA_TQ-AlignFAST-VQA
FastH3 v2 (VSA-H3)26.0078.4290.7179.27
VDN-H328.0389.2093.1383.41

We have additionally compared SLA2-H3 and VDN-H3 under the same training compute budget, using only a fraction of the compute required for full training. As shown below, VDN-H3 achieves higher scores on all four reported quality metrics: VQA_T improves from 75.91 to 86.66, and FAST-VQA from 77.44 to 81.74. This extends our evaluation to a trained sparse–linear baseline.

MethodVQA_AVQA_TQ-AlignFAST-VQA
Dense H3 (reference)27.5488.2092.2282.68
SLA2-H324.1275.9188.1577.44
VDN-H3 (Stage B only)28.1686.6691.9281.74

ARL² and ReHyAt share the principle of combining local Softmax attention with recurrent memory. VDN’s added value lies in three coordinated design choices. First, its frame-level joint update accounts for correlations among spatial tokens, reducing interference between their memory writes. Second, its attention pattern combines local cross-frame Softmax attention, bidirectional long-range memory, and global first/last-frame anchors to retain complementary temporal context. Third, normalization and branch-specific gates and output projections calibrate the Softmax and linear contributions, helping preserve quality when converting pretrained attention. We now discuss ARL² and ReHyAt in the Related Work section of the revised version (lines 509–516). All newly added or modified text in the revised paper is highlighted in red.

Weakness 3: Ablations of design choices

Some important design choices are not ablated: Most importantly, there is no forward-only variant showing the contribution of the reverse linear scan. This also matters for deployment because the reverse scan requires access to future latent chunks and prevents causal chunk-wise generation. Furthermore, the proposed three-stage adaptation pipeline is also not ablated with respect to the final quality. The parameter displacement analysis does not show if A1, A2, and the LoRA-based stage B are all needed; another aspect that I didn't find to be directly motivated intuitively or empirically is the use of LoRA for the tuning in stage B.

The current version of VDN is designed primarily for bidirectional video generation. We therefore maintain symmetry between the forward and reverse linear scans, so that temporal context can be incorporated from both directions during denoising. However, we believe that for autoregressive generation, the reverse linear scan can be removed without issue.

We have added a comparison of Stage-B-only adaptation and A1 → A2 → Stage B under the same total training budget. The staged pipeline improves three of the four reported quality metrics. This directly supports the combined benefit of A1/A2 pre-alignment under the tested budget.

MethodVQA_AVQA_TQ-AlignFAST-VQA
Stage B only28.1686.6691.9281.74
A1 + A2 + Stage B28.0187.3592.3582.23
Staged − B only-0.15+0.69+0.43+0.49

Our choice of LoRA in Stage B is motivated primarily by compatibility with few-step LoRA adapters. The attention-adaptation and few-step updates can be composed on the same backbone, keeping architecture adaptation modular. In our experiments, directly applying a compatible few-step LoRA together with the Stage-B LoRA already produces reasonable video quality without additional DMD distillation. This provides a practical way to connect attention conversion to few-step generation. More broadly, we want this interface to remain reusable, so that future, better-trained few-step LoRAs for the same backbone can also be incorporated.

Question 2: Instruction following before and after step distillation

Instruction following before and after step distillation: At 50 NFE, VDN-H3 is lower than Dense H3 and Radial Attention on FIRM instruction following, while the 8 NFE model appears to recover strongly on this metric. I am curious whether this improvement comes from DMD distillation, initialization from the H3-Turbo LoRA, or another part of the training pipeline. This could clarify whether the 50 NFE result reflects the attention architecture or its adaptation.

The improvement comes mainly from the H3-Turbo LoRA initialization; DMD distillation adds a smaller gain on top.

To separate the two, we applied the third-party H3-Turbo LoRA directly to each model, with no additional training, and sampled at 8 NFE. As shown below, the LoRA alone lowers FIRM instruction following for Dense H3 (2.25 to 2.13) but raises it for VDN-H3 (2.12 to 2.21), so VDN-H3 already scores above Dense H3 before any distillation. DMD distillation, which starts from this LoRA, then brings VDN-H3 to 2.25.

SettingDense H3VDN-H3
50 NFE2.252.12
8 NFE, H3-Turbo LoRA applied without training2.132.21
8 NFE, after DMD distillation—2.25

Of the 0.13 total gain for VDN-H3, 0.09 therefore comes from the LoRA and 0.04 from DMD. This indicates that the lower score at 50 NFE is specific to that setting rather than a limitation of the attention architecture: with the same few-step LoRA, the same VDN-H3 backbone scores above Dense H3, and after distillation it matches the 50-NFE Dense H3 score.

Question 3: Teacher selection for step distillation

Teacher selection for step distillation: The 8 NFE student is distilled from the already converted 50 NFE VDN H3 model rather than directly from Dense H3. I understand that this separates architecture conversion from step reduction. However, I wonder whether using Dense H3 as the teacher could help recover some of the quality lost during architecture conversion. Was this alternative tested or considered?

We considered and tested Dense H3 as the teacher in small-scale experiments. This requires DMD to bridge the gaps from both architecture conversion and few-step distillation simultaneously, making optimization more challenging. In these experiments, we observed less visible noise in generated videos when using 50-NFE VDN-H3 as the teacher than when using 50-NFE Dense H3. We therefore adopted VDN H3 as the teacher, separating architecture conversion from step reduction.

We appreciate the reviewer's suggestions, which have strengthened the paper, and we welcome any further questions.

Thank you for the detailed rebuttal and the additional experiments. The response addresses most of my concerns, in particular the lack of human evaluation, the limited coverage of efficient and related hybrid attention methods, and the questions around instruction following and the distillation teacher.

The human evaluation is particularly helpful. While the preference is slightly toward Dense H3, the large fraction of ties makes the claim that VDN largely preserves the quality of the original model considerably more convincing. The added VSA/SLA2 results also make the efficiency and quality comparison stronger.

My main remaining concern is the role of the reverse linear scan. The rebuttal explains the motivation for bidirectional memory, but its contribution is still not isolated experimentally. In particular, it remains unclear how much of the reported quality depends on the reverse pathway, while this choice also means that future latent chunks need to be available during inference.

Overall, the rebuttal addresses most of the issues I raised, and I feel more confident in my positive assessment of the paper.

Thank you for the encouraging assessment. Due to the time limit of the rebuttal period, we removed the reverse scan from the trained VDN-H3 checkpoint and reran inference without retraining.

As shown below, removing the reverse scan lowers VQA_T from 87.61 to 86.84 and also lowers FAST-VQA and MaxVQA clear-motion, while VQA_A and Q-Align remain essentially unchanged. The largest drops are in the temporal metrics, indicating that the trained model relies on the reverse pathway mainly for temporal quality.

MethodVQA_AVQA_TQ-AlignFAST-VQAMaxVQA clear-motion
VDN-H327.6987.6192.1482.3482.74
VDN-H3 w/o reverse scan27.6386.8492.3482.1982.31

Regarding the availability of future latent chunks, the reverse scan adds no requirement beyond that of the base model. VDN-H3 replaces the full attention of Dense H3, which denoises all latent frames jointly and already lets each frame attend to future frames; the reverse scan gives the linear memory the same bidirectional context. Causal generation, by design, does not use future information, so in that setting we expect the reverse scan to be omitted and only the forward scan kept.

We will add this ablation and the discussion of causal generation to the revised paper. Thank you again for your support.

Reviewer_2C8FOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Video DeltaNet combines local softmax attention and boundary anchors with bidirectional linear memory. Its frame-wise Video Delta Attention update jointly resolves spatial writes through a regularized least-squares objective. Staged adaptation, few-step distillation, and optimized kernels enable efficient deployment on a pretrained video-generation backbone.

Critical strengths
  1. The motivation is clear and has mathematical motivation.

  2. The non-expansiveness guarantee is appropriately restricted to inherited-state transitions with fixed features and gates.

Critical weaknesses
  1. The empirical advantage of the proposed update over a simpler additive update is mixed. Table 4 shows improvements on several metrics, but the additive variant performs better on Q-Align and DOVER++. A matched comparison of quality, runtime, memory, and numerical stability would help establish whether the joint solve provides a consistently favorable trade-off.
  2. The necessity of the staged adaptation procedure is not directly established. Table 3 measures parameter displacement across stages. This characterizes optimization behavior, but does not demonstrate that A1 or A2 is necessary. Budget-matched removal or replacement experiments would be more informative.
  3. Acceleration claims require consistent resource accounting. The same-GPU, same-NFE B200 backbone gain is approximately 2.6×. The larger 119.3× deployment figure additionally includes fewer denoising evaluations and an increase from one to eight GPUs. The paper provides this decomposition, but the architectural contribution should primarily be assessed using matched-resource measurements.
  4. Quality preservation is evaluated on a relatively limited prompt set. The 103-prompt evaluation and the appendix’s paired significance annotations are useful, but broader coverage of occlusion, identity persistence, complex motion, and long-range interactions would strengthen the conclusion. Human paired comparisons would also complement the automated metrics.
  5. The configuration underlying the architecture ablations is unclear. The VDN VQA-A score in Table 4 differs substantially from the principal 50-NFE comparison. This may reflect a different evaluation configuration, but the prompt set, checkpoint, NFE, and random seeds should be stated explicitly.
Questions for authors
  1. Responses to the weaknesses.

  2. And also:

  • What are the quality, runtime, and stability differences between the joint solve and additive update under matched settings?
  • How much performance is lost when A1 or A2 is removed under an equal training budget?
  • What explains the difference between the architecture-ablation scores and the main evaluation scores?
4 comments

We sincerely thank Reviewer 2C8F for the careful reading and the constructive, technically precise comments. We respond to each point below; since Questions 1–3 overlap with Weaknesses 1, 2 and 5, we answer them together.

Weakness 1 & Question 1: Joint solve versus additive update

Weakness 1. The empirical advantage of the proposed update over a simpler additive update is mixed. Table 4 shows improvements on several metrics, but the additive variant performs better on Q-Align and DOVER++. A matched comparison of quality, runtime, memory, and numerical stability would help establish whether the joint solve provides a consistently favorable trade-off.

Question 1. What are the quality, runtime, and stability differences between the joint solve and additive update under matched settings?

The matched results support the joint update as the better overall choice for VDN: it improves five of eight quality metrics and matches FIRM perceptual quality. Its gains span visual quality, instruction following, and world coherence, as shown in the full comparison below.

MethodVQA_AVQA_TQ-AlignFAST-VQADOVER++FIRM IFFIRM PQFIRM WC
Scaled additive30.1489.4292.8883.323.762.254.471.76
Joint30.5690.1192.6983.473.622.274.471.83

These gains require little extra compute and no additional peak inference memory. A full FP8 inference forward pass takes 11.26 s with the joint update versus 11.05 s with the additive update, only 1.9% more, while both use 25.25 GiB of peak memory. The attention block’s training forward and backward takes 664 ms versus 638 ms, an increase of 4.1%. This makes the quality improvements affordable in practice.

MethodFP8 forward (s)Peak memory (GiB)Scan only (ms)Training fwd + bwd (ms)
Scaled additive11.0525.255.1638
Joint11.2625.2511.0664

The joint solve gives us a further design advantage: its write strength adapts to the correlations among spatial keys in each frame, while retaining a non-expansive inherited-state transition without an extra frame-size correction. Appendix A shows that, for fixed frame features and gates, the eigenvalues of (I + Aₜ)⁻¹ lie in (0, 1], so the inherited state is not amplified by this transition. The scaled additive baseline requires additional 1/√U key scaling to obtain this non-expansive property.

Weakness 2 & Question 2: Necessity of the staged adaptation

Weakness 2. The necessity of the staged adaptation procedure is not directly established. Table 3 measures parameter displacement across stages. This characterizes optimization behavior, but does not demonstrate that A1 or A2 is necessary. Budget-matched removal or replacement experiments would be more informative.

Question 2. How much performance is lost when A1 or A2 is removed under an equal training budget?

We have added a comparison of Stage-B-only adaptation and A1 → A2 → Stage B under the same total training budget. The staged pipeline improves three of the four reported quality metrics. This directly supports the combined benefit of A1/A2 pre-alignment under the tested budget.

MethodVQA_AVQA_TQ-AlignFAST-VQA
Stage B only28.1686.6691.9281.74
A1 + A2 + Stage B28.0187.3592.3582.23
Staged − B only-0.15+0.69+0.43+0.49

Weakness 3: Resource accounting for the acceleration claims

Acceleration claims require consistent resource accounting. The same-GPU, same-NFE B200 backbone gain is approximately 2.6×. The larger 119.3× deployment figure additionally includes fewer denoising evaluations and an increase from one to eight GPUs. The paper provides this decomposition, but the architectural contribution should primarily be assessed using matched-resource measurements.

Thank you for the suggestion. We have revised the abstract (lines 23–26), the introduction (lines 60–62), and the experimental section (Section 7.3, lines 426–427 and 462–463, and the caption of Table 2) to more clearly emphasize the backbone speedup measured under matched GPU resources and NFE. All newly added or modified text in the revised paper is highlighted in red.

Weakness 4: Coverage of the quality evaluation

Quality preservation is evaluated on a relatively limited prompt set. The 103-prompt evaluation and the appendix’s paired significance annotations are useful, but broader coverage of occlusion, identity persistence, complex motion, and long-range interactions would strengthen the conclusion. Human paired comparisons would also complement the automated metrics.

To broaden coverage, we created three new prompt sets, one each for complex motion, identity persistence, and long-range interactions. The prompts were generated with Claude Opus 5.5 and then enhanced with H3's native Context IR prompt enhancement. Each set is evaluated at 8 NFE with the metric that matches its category: Motion Smoothness on the complex-motion set, and Subject Consistency on the identity-persistence and long-range-interaction sets. These prompts target more challenging scenarios than the standard VBench prompts, so the absolute scores are lower than the VBench results in our response to Reviewer QfAT. Across all three categories, VDN-H3 preserves the quality of Dense H3: it is slightly higher on identity persistence and long-range interactions, and close to Dense H3 in motion smoothness on complex-motion prompts.

Method (8 NFE)Complex motion (Motion Smoothness)Identity persistence (Subject Consistency)Long-range interactions (Subject Consistency)
Dense H397.2469.9274.25
VDN-H397.1470.6374.85

We have added a method-blind, pairwise human preference study with 10 evaluators and 103 video pairs. Model names were hidden; evaluators selected either video or a tie to record overall preference. At 50 NFE, most judgments are ties, suggesting similar perceived quality between VDN-H3 and Dense H3.

JudgmentsDense H3 50 NFETieVDN-H3 50 NFE
All judgments22.3%59.2%18.4%

Weakness 5 & Question 3: Configuration of the architecture ablations

Weakness 5. The configuration underlying the architecture ablations is unclear. The VDN VQA-A score in Table 4 differs substantially from the principal 50-NFE comparison. This may reflect a different evaluation configuration, but the prompt set, checkpoint, NFE, and random seeds should be stated explicitly.

Question 3. What explains the difference between the architecture-ablation scores and the main evaluation scores?

We trained the architecture-ablation variants for a relatively small number of steps because fully training every variant would require substantial compute. All variants and baselines in Table 4 were trained with matched compute resources. Therefore, the scores in Table 4 support comparisons within this controlled setting and should not be directly compared with the main evaluation scores obtained after full training.

We thank the reviewer again for the valuable feedback and are happy to provide further details during the discussion.

Reviewer_2C8FOpenReview ↗

Responses to rebuttal

The results stengthen the evidence that the method strikes a reasonable balance betwee nquality and computational cost, however, the independent necessity of each training stage, epmprical stability, and complex interactio capablities are still not fully established. THe reviewer tend to keep the score.

Thank you for your constructive feedback. We will include additional results in the camera-ready version to further demonstrate our contributions.

Reviewer_QfATOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Video DeltaNet (VDN), a hybrid attention architecture adapted from the pretrained MiniMax H3 model. It divides video attention into two complementary branches:

Softmax attention preserves detailed interactions within neighboring temporal chunks and direct access to the first and last latent frames. Interactions involving text or audio also retain Softmax attention.

Bidirectional linear memory to replace expensive attention between distant video frames with a compact memory representation, while preserving conventional Softmax attention for nearby frames.

Critical strengths

The proposed Video Delta Attention (VDA) introduces a joint frame-wise memory update that accounts for correlations among spatial tokens, mitigating potentially conflicting or excessive updates arising from independent patch-wise corrections. The ablation results support the effectiveness of this design, with improvements over the scaled additive alternative on five of eight quality metrics, although performance is lower on two and unchanged on one.

Critical weaknesses
  1. Unclear presentation and organization: The paper is difficult to follow, particularly in the introduction, which delves into architectural details before clearly establishing the main idea and its underlying intuition. For example, the explanation in lines 75–77 is unnecessarily complicated. A preliminary section introducing the essential concepts and notation would significantly improve readability. Similar presentation issues persist throughout the paper. In particular, Figures 4 and 5 lack units (probably the range of the measures), making the results difficult to interpret without consulting the main text.

  2. Unclear attribution of the performance gains and additional training complexity: VDN is not training-free; it requires several adaptation stages, including layer-wise alignment, end-to-end alignment, and LoRA co-adaptation, followed by few-step distillation and optimized GPU kernels. Although Table 2 separates the contributions of several acceleration stages, the overall performance gains combine architectural improvements, distillation, and also hardware-specific optimizations. It is not clear how the hardware optimization and distributed computing contirbute. Hardware optimization contibution is somewhat orthjogonal to the algorithm and can be also implemented in compared and existing methods.

  3. Methodological novelty and insufficient comparisons with related work: Previous work, including FAST-AR [1], has demonstrated that correlations and redundancy in video attention representations can be exploited to reduce computation. Although VDN introduces a different mechanism, the authors should better clarify its conceptual novelty relative to such approaches. Moreover, the experimental comparisons focus primarily on dense and sparse attention methods, without evaluating recent hybrid Softmax–linear approaches such as Attention Surgery (CVPR 2026). A matched comparison would better establish VDN's specific contribution.

  4. Modest and mixed quality improvements: The incremental benefit of the proposed VDN is not consistently convincing. In Table 4, the joint update improves five of eight quality metrics over the scaled additive alternative, but performs worse on two (and ties on one). Similarly, Figure 4 shows that VDN outperforms Radial Attention on six of seven shared quality metrics at nearly identical inference times, but most improvements are relatively modest, while Instruction Following deteriorates. It remains unclear whether these differences translate into meaningful perceptual improvements. Statistical significance analysis and particularly a human preference study would strengthen these conclusions.

  5. Insufficient evaluation: The evaluation relies on only 103 fixed third-party prompts, without clearly identifying their source, selection criteria, or diversity in terms of scene complexity, motion, and temporal requirements. The prompts and complete sampling protocol are not provided. Furthermore, the absence of standardized benchmarks such as VBench or VBench 2.0 makes it difficult to assess performance across established quality dimensions and compare against recent methods. Given the relatively modest improvements over competing approaches, a larger and more transparent evaluation is necessary. The paper lacks a side by side video comparison. Such paper must show many such results to allow a clear performance analysis (snapshots are not enough).

  6. Limited generality and quality-efficiency analysis: The method is evaluated only on MiniMax H3, at a single resolution and approximately 14-second video duration. Although Tables 2 and 3 provide efficiency and training analyses, neither reports video quality alongside the measured speedups. While quality is evaluated separately in Figures 4 and 5. A unified quality-versus-latency analysis across the different acceleration stages is needed.

In summary, although the paper has some merits and demonstrates improvements in certain performance metrics, it suffers from several significant shortcomings, including unclear presentation, an inadequately described evaluation set, the absence of established benchmarks such as VBench, lack of video qualitative comaprisons, and inconclusive experimental results. Taken together, these limitations leave the paper below the acceptance bar, and I therefore recommend rejection.

[1] D. Samuel et al., FAST-AR: Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention, ICML 2026.

Questions for authors

Please see weaknesses

5 comments

We thank Reviewer QfAT for the detailed review. We have addressed each weakness below and revised the paper, in which all newly added or modified text is highlighted in red.

Weakness 1: Presentation and organization

Unclear presentation and organization: The paper is difficult to follow, particularly in the introduction, which delves into architectural details before clearly establishing the main idea and its underlying intuition. For example, the explanation in lines 75–77 is unnecessarily complicated. A preliminary section introducing the essential concepts and notation would significantly improve readability. Similar presentation issues persist throughout the paper. In particular, Figures 4 and 5 lack units (probably the range of the measures), making the results difficult to interpret without consulting the main text.

We have reorganized the introduction around the central intuition: preserve direct Softmax interactions for nearby and boundary frames, and summarize distant video context with bidirectional memory. VDA updates this memory jointly from all spatial tokens in a frame, accounting for correlations among their writes. Details are in Section 1, lines 40–58, and the new Section 2, lines 108–141.

We have also revised Figures 4 and 5: each panel now states the metric's range or unit directly under its label. The captions (lines 391–393 and 407–408) and the Metrics paragraph of Section 7.1 (lines 365–368) note the truncated axes and distinguish quality scores from the motion-magnitude diagnostic. All newly added or modified text in the revised paper is highlighted in red.

Weakness 2: Attribution of the speedup and training complexity

Unclear attribution of the performance gains and additional training complexity: VDN is not training-free; it requires several adaptation stages, including layer-wise alignment, end-to-end alignment, and LoRA co-adaptation, followed by few-step distillation and optimized GPU kernels. Although Table 2 separates the contributions of several acceleration stages, the overall performance gains combine architectural improvements, distillation, and also hardware-specific optimizations. It is not clear how the hardware optimization and distributed computing contirbute. Hardware optimization contibution is somewhat orthjogonal to the algorithm and can be also implemented in compared and existing methods.

We would like to point out that the staged training pipeline is part of our contribution rather than an additional burden: it is designed to adapt the converted architecture effectively under a limited training budget. Under the same total training budget, A1 → A2 → Stage B achieves the best VQA_T, Q-Align, and FAST-VQA scores among the compared adaptation strategies, as shown below. This shows that the staged design makes more effective use of the available training compute.

MethodVQA_AVQA_TQ-AlignFAST-VQA
Stage B only28.1686.6691.9281.74
A1 + A2 + Stage B28.0187.3592.3582.23
Staged − B only-0.15+0.69+0.43+0.49

Regarding the attribution of the speedup, we would like to point out that Table 2 already reports the three sources separately: 2.6× from the backbone at the same GPU count and NFE, a further 6.3× from few-step distillation, and 7.4× from eight-GPU execution. Following the reviewer's comment, we have made this attribution more explicit in the abstract (lines 23–26), the introduction (lines 60–62), and Section 7.3 (lines 426–427 and 462–463). These changes are highlighted in red.

We respectfully disagree that hardware optimization is orthogonal to the algorithm in this setting. The per-block latency breakdown below, measured on one B200, separates what is generic from what is specific to our method.

ConfigurationPer-block latency (ms)Speedup over Dense H3
Dense H3320.91.00×
+ FP8 linear layers and fused kernels256.41.25×
+ hybrid attention, without dedicated kernels287.91.11×
+ hybrid attention, with dedicated kernels122.12.63×

The hybrid attention with its dedicated kernels accounts for 2.1× of the 2.6×. This gain is measured on top of a Dense H3 that already includes the generic optimizations (FP8 linear layers and fused kernels, 1.25×), so it comes from the method rather than from generic engineering.

The dedicated kernels are part of the method. Without them, the hybrid attention is not faster (287.9 ms versus 256.4 ms), because dense Softmax attention already runs on heavily optimized kernels such as FlashAttention-4, whereas a new operator does not. Our kernels implement VDA-specific operators, including frame statistics, boundary-state gathering, and the joint solve. For the same reason, Radial Attention and FastH3 v2 rely on their own sparse-attention kernels, and we used these optimized backends in our evaluation for a fair comparison.

Weakness 3: Novelty and comparison with related work

Methodological novelty and insufficient comparisons with related work: Previous work, including FAST-AR [1], has demonstrated that correlations and redundancy in video attention representations can be exploited to reduce computation. Although VDN introduces a different mechanism, the authors should better clarify its conceptual novelty relative to such approaches. Moreover, the experimental comparisons focus primarily on dense and sparse attention methods, without evaluating recent hybrid Softmax–linear approaches such as Attention Surgery (CVPR 2026). A matched comparison would better establish VDN's specific contribution.

We now cite FAST-AR and Attention Surgery in the Related Work section of the revised version (lines 509–516, highlighted in red). FAST-AR reduces autoregressive inference cost through temporal KV-cache compression and sparse attention. Attention Surgery converts pretrained video transformers using uniformly sampled Softmax tokens and learned kernelized linear attention. The revised text states how VDN differs in its memory update and context allocation.

We would like to point out that VDN's novelty goes beyond the general observation that video attention contains redundancy. It lies in three contributions.

A joint frame-wise memory update. VDA solves a regularized least-squares problem over all spatial writes, giving Sₜ = (S̄ₜ + Bₜ)(I + Aₜ)⁻¹. The inverse accounts for within-frame key correlations and yields a non-expansive inherited-state transition at fixed features and gates without additional frame-size key scaling.

An allocation between Softmax and linear attention that is validated by ablations. Softmax handles a local temporal window and two boundary anchors, while bidirectional linear memory handles the remaining context. In Table 4, removing the anchors lowers seven of eight quality metrics, and removing the linear branch lowers six of eight.

A staged training pipeline for converting pretrained models. Under the same training budget, the staged pipeline improves three of four quality metrics over direct Stage-B adaptation, as shown in our response to Weakness 2.

Since we could not find a publicly available training implementation of Attention Surgery, we instead evaluate SLA2, another hybrid method combining sparse Softmax attention with linear attention. We compare SLA2-H3 and VDN-H3 under the same partial training compute budget. As shown below, VDN-H3 achieves higher scores on all four reported quality metrics, with VQA_T increasing from 75.91 to 86.66 and FAST-VQA from 77.44 to 81.74.

MethodVQA_AVQA_TQ-AlignFAST-VQA
Dense H3 (reference)27.5488.2092.2282.68
SLA2-H324.1275.9188.1577.44
VDN-H3 (Stage B only)28.1686.6691.9281.74

This comparison with a trained Softmax–linear hybrid baseline complements the joint-versus-additive ablation and further supports the effectiveness of VDN’s design.

Weakness 4: Quality improvements and perceptual evidence

Modest and mixed quality improvements: The incremental benefit of the proposed VDN is not consistently convincing. In Table 4, the joint update improves five of eight quality metrics over the scaled additive alternative, but performs worse on two (and ties on one). Similarly, Figure 4 shows that VDN outperforms Radial Attention on six of seven shared quality metrics at nearly identical inference times, but most improvements are relatively modest, while Instruction Following deteriorates. It remains unclear whether these differences translate into meaningful perceptual improvements. Statistical significance analysis and particularly a human preference study would strengthen these conclusions.

We have added a method-blind, pairwise human preference study with 10 evaluators and 103 video pairs per model comparison. Model names were hidden; evaluators selected either video or a tie to record overall preference.

JudgmentsDense H3 50 NFETieVDN-H3 50 NFE
All judgments22.3%59.2%18.4%

At 50 NFE, most judgments are ties, suggesting similar perceived quality between VDN-H3 and Dense H3.

We also compared VDN-H3 with Radial Attention at 50 NFE, where the two methods have matched latency. Evaluators prefer VDN-H3 in 43.2% of judgments and Radial Attention in 16.7%, with 40.1% ties. This shows that VDN-H3's advantage over Radial Attention on the automated metrics translates into a perceptible quality improvement.

JudgmentsVDN-H3 50 NFETieRadial Attention 50 NFE
All judgments43.2%40.1%16.7%

Weakness 5: Evaluation protocol and benchmarks

Insufficient evaluation: The evaluation relies on only 103 fixed third-party prompts, without clearly identifying their source, selection criteria, or diversity in terms of scene complexity, motion, and temporal requirements. The prompts and complete sampling protocol are not provided. Furthermore, the absence of standardized benchmarks such as VBench or VBench 2.0 makes it difficult to assess performance across established quality dimensions and compare against recent methods. Given the relatively modest improvements over competing approaches, a larger and more transparent evaluation is necessary. The paper lacks a side by side video comparison. Such paper must show many such results to allow a clear performance analysis (snapshots are not enough).

We have added VBench evaluation using the official VBench prompts. For 345-frame videos, we compare Dense H3, Radial Attention, and VDN-H3 at 50 NFE, and Dense H3, FastH3 v2, and VDN-H3 at 8 NFE. The table reports five VBench dimensions with their arithmetic mean.

Method (345 frames)Subject ConsistencyBackground ConsistencyMotion SmoothnessAesthetic QualityImaging QualityAverage
Dense 5095.3595.4999.3763.5472.1185.17
Radial 5095.1295.3099.1260.5771.8884.40
VDN 5096.4296.5399.4064.5571.4985.68
Dense 894.8495.6699.3263.5972.4085.16
FastH3 v2 894.2994.8298.6562.8374.7085.06
VDN 896.3296.5599.3764.8872.0385.83

These results demonstrate VDN’s strong visual quality and temporal consistency in both standard and few-step video generation.

We have also added ten side-by-side videos to the supplementary materials. Each video compares Dense H3, FastH3 v2, and VDN-H3 from left to right, all at 8 NFE and using the same prompt. The videos allow direct inspection of motion, temporal consistency, and instruction following.

Weakness 6: Generality and quality–efficiency analysis

Limited generality and quality-efficiency analysis: The method is evaluated only on MiniMax H3, at a single resolution and approximately 14-second video duration. Although Tables 2 and 3 provide efficiency and training analyses, neither reports video quality alongside the measured speedups. While quality is evaluated separately in Figures 4 and 5. A unified quality-versus-latency analysis across the different acceleration stages is needed.

We have added a unified quality-versus-latency comparison across the acceleration stages. The table below reports single-B200 denoising latency for the 14.3-second, 768p workload together with the quality metrics reported in the paper.

ConfigurationNFELatency (s)SpeedupVQA_AVQA_TQ-AlignFAST-VQA
Dense H350799.61.0×27.5488.2092.2282.68
VDN-H350307.92.6×27.6987.6192.1482.34
VDN-H3 + few-step849.316.2×28.0389.2093.1383.41

We have also extended the quality evaluation to two shorter video settings, 141 and 243 frames, complementing the 345-frame evaluation. The tables below compare Dense H3, Radial Attention, and VDN-H3 at 50 NFE, and Dense H3, FastH3 v2, and VDN-H3 at 8 NFE. We report five VBench dimensions and their arithmetic mean in the Average column.

Method (141 frames)Subject ConsistencyBackground ConsistencyMotion SmoothnessAesthetic QualityImaging QualityAverage
Dense 5096.8596.8999.4464.6872.5186.07
Radial 5096.7496.6699.2761.5371.8585.21
VDN 5097.9797.4699.4865.7172.7286.67
Dense 897.2696.8899.4365.2472.1186.18
FastH3 v2 894.5795.3898.7164.0274.1685.37
VDN 898.0997.1799.4765.1573.0286.58
Method (243 frames)Subject ConsistencyBackground ConsistencyMotion SmoothnessAesthetic QualityImaging QualityAverage
Dense 5094.9795.5999.2963.7873.0085.33
Radial 5093.9095.0099.0460.4271.1583.90
VDN 5095.7396.0799.3663.8772.4385.49
Dense 894.4295.2199.2664.0972.4885.09
FastH3 v2 891.1193.5098.6462.3174.5084.01
VDN 895.1995.6599.3164.3673.1485.53

These results demonstrate that VDN’s quality benefits extend across video durations in both standard and few-step generation.

We hope these additions address the reviewer's concerns, and we would be glad to clarify any remaining points during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.