- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this paper the authors presents Video DeltaNet, a hybrid attention method for accelerating video diffusion models. The proposed method uses Softmax attention for nearby frames and global boundary frames, while distant video context is represented using bidirectional linear memory. The proposed Video Delta Attention updates the memory jointly for all spatial tokens in a frame, accounting for correlations between their writes. The paper also introduces a staged procedure for adapting the pretrained MiniMax H3 model, followed by a step distillation and several kernel optimizations. The experiments show notable practical speedups on H200 and B200 GPUs, while the reported automatic quality metrics remain generally close to those of the original Dense H3 model on most of the metrics.
Critical strengths
-
The frame-wise delta update is well motivated: The method jointly handles all spatial tokens of a frame and accounts for correlations between their writes. This is more natural for video than applying token-wise delta updates using an arbitrary spatial order and is backed up by observed advantages in empirical evaluation.
-
The paper reports strong practical speedups: The authors provide full backbone latency on both H200 and B200, scaling results for different video lengths, and detailed kernel ablations. The reported speedups are therefore supported by actual system measurements, rather than only theoretical complexity.
-
The hybrid design is practical: Local interactions remain under Softmax attention, while distant context is handled by linear memory. The first and last frames are also kept as global anchors. This seems like a reasonable way to reduce attention cost while preserving important local and boundary information.
-
The paper presents a complete conversion and deployment pipeline: It covers architecture adaptation, step distillation, Triton kernel optimization, and multi-GPU execution. This makes the work more useful than a method evaluated only at the operator level.
Critical weaknesses
-
The visual quality evaluation is not sufficient: While automatic benchmark metrics surely bring values to the empirical evaluations, their limitations in matching human perceptual preference, which I believe is the ultimate goal for the video generation, is known to the community. The paper offers no human evaluation studies. This is important because preserving the visual and temporal quality of Dense H3 is one of the main claims.
-
The baseline comparison is narrow: At 50 NFE, the main efficient baseline is Radial Attention, which is a training-free efficient attention method, while VDN uses staged training. The experimental comparison would be stronger with a few training-based efficient attention baselines, e.g. VSA, SLA, SLA2, etc.. The paper also does not discuss some of the closely related hybrid recurrent methods e.g. ARL2 and ReHyAt, both of which combine local Softmax attention with recurrent linear memory for longer range video context. A discussion would help clarify the novelty, distinction and the extra value of the proposed hybrid design over the existing works.
-
Some important design choices are not ablated: Most importantly, there is no forward-only variant showing the contribution of the reverse linear scan. This also matters for deployment because the reverse scan requires access to future latent chunks and prevents causal chunk-wise generation. Furthermore, the proposed three-stage adaptation pipeline is also not ablated with respect to the final quality. The parameter displacement analysis does not show if A1, A2, and the LoRA-based stage B are all needed; another aspect that I didn't find to be directly motivated intuitively or empirically is the use of LoRA for the tuning in stage B.
Questions for authors
-
Subjective evaluation: Could the authors provide a method-blind human preference study between VDN-H3 and Dense H3? This evidence can directly affect my assessment of the quality-preservation claim, even if done at a moderate scale.
-
Instruction following before and after step distillation: At 50 NFE, VDN-H3 is lower than Dense H3 and Radial Attention on FIRM instruction following, while the 8 NFE model appears to recover strongly on this metric. I am curious whether this improvement comes from DMD distillation, initialization from the H3-Turbo LoRA, or another part of the training pipeline. This could clarify whether the 50 NFE result reflects the attention architecture or its adaptation.
-
Teacher selection for step distillation: The 8 NFE student is distilled from the already converted 50 NFE VDN H3 model rather than directly from Dense H3. I understand that this separates architecture conversion from step reduction. However, I wonder whether using Dense H3 as the teacher could help recover some of the quality lost during architecture conversion. Was this alternative tested or considered?
4 comments
Weakness 3: Ablations of design choices
Some important design choices are not ablated: Most importantly, there is no forward-only variant showing the contribution of the reverse linear scan. This also matters for deployment because the reverse scan requires access to future latent chunks and prevents causal chunk-wise generation. Furthermore, the proposed three-stage adaptation pipeline is also not ablated with respect to the final quality. The parameter displacement analysis does not show if A1, A2, and the LoRA-based stage B are all needed; another aspect that I didn't find to be directly motivated intuitively or empirically is the use of LoRA for the tuning in stage B.
The current version of VDN is designed primarily for bidirectional video generation. We therefore maintain symmetry between the forward and reverse linear scans, so that temporal context can be incorporated from both directions during denoising. However, we believe that for autoregressive generation, the reverse linear scan can be removed without issue.
We have added a comparison of Stage-B-only adaptation and A1 → A2 → Stage B under the same total training budget. The staged pipeline improves three of the four reported quality metrics. This directly supports the combined benefit of A1/A2 pre-alignment under the tested budget.
| Method | VQA_A | VQA_T | Q-Align | FAST-VQA |
|---|---|---|---|---|
| Stage B only | 28.16 | 86.66 | 91.92 | 81.74 |
| A1 + A2 + Stage B | 28.01 | 87.35 | 92.35 | 82.23 |
| Staged − B only | -0.15 | +0.69 | +0.43 | +0.49 |
Our choice of LoRA in Stage B is motivated primarily by compatibility with few-step LoRA adapters. The attention-adaptation and few-step updates can be composed on the same backbone, keeping architecture adaptation modular. In our experiments, directly applying a compatible few-step LoRA together with the Stage-B LoRA already produces reasonable video quality without additional DMD distillation. This provides a practical way to connect attention conversion to few-step generation. More broadly, we want this interface to remain reusable, so that future, better-trained few-step LoRAs for the same backbone can also be incorporated.
Question 2: Instruction following before and after step distillation
Instruction following before and after step distillation: At 50 NFE, VDN-H3 is lower than Dense H3 and Radial Attention on FIRM instruction following, while the 8 NFE model appears to recover strongly on this metric. I am curious whether this improvement comes from DMD distillation, initialization from the H3-Turbo LoRA, or another part of the training pipeline. This could clarify whether the 50 NFE result reflects the attention architecture or its adaptation.
The improvement comes mainly from the H3-Turbo LoRA initialization; DMD distillation adds a smaller gain on top.
To separate the two, we applied the third-party H3-Turbo LoRA directly to each model, with no additional training, and sampled at 8 NFE. As shown below, the LoRA alone lowers FIRM instruction following for Dense H3 (2.25 to 2.13) but raises it for VDN-H3 (2.12 to 2.21), so VDN-H3 already scores above Dense H3 before any distillation. DMD distillation, which starts from this LoRA, then brings VDN-H3 to 2.25.
| Setting | Dense H3 | VDN-H3 |
|---|---|---|
| 50 NFE | 2.25 | 2.12 |
| 8 NFE, H3-Turbo LoRA applied without training | 2.13 | 2.21 |
| 8 NFE, after DMD distillation | — | 2.25 |
Of the 0.13 total gain for VDN-H3, 0.09 therefore comes from the LoRA and 0.04 from DMD. This indicates that the lower score at 50 NFE is specific to that setting rather than a limitation of the attention architecture: with the same few-step LoRA, the same VDN-H3 backbone scores above Dense H3, and after distillation it matches the 50-NFE Dense H3 score.
Question 3: Teacher selection for step distillation
Teacher selection for step distillation: The 8 NFE student is distilled from the already converted 50 NFE VDN H3 model rather than directly from Dense H3. I understand that this separates architecture conversion from step reduction. However, I wonder whether using Dense H3 as the teacher could help recover some of the quality lost during architecture conversion. Was this alternative tested or considered?
We considered and tested Dense H3 as the teacher in small-scale experiments. This requires DMD to bridge the gaps from both architecture conversion and few-step distillation simultaneously, making optimization more challenging. In these experiments, we observed less visible noise in generated videos when using 50-NFE VDN-H3 as the teacher than when using 50-NFE Dense H3. We therefore adopted VDN H3 as the teacher, separating architecture conversion from step reduction.
We appreciate the reviewer's suggestions, which have strengthened the paper, and we welcome any further questions.
Thank you for the detailed rebuttal and the additional experiments. The response addresses most of my concerns, in particular the lack of human evaluation, the limited coverage of efficient and related hybrid attention methods, and the questions around instruction following and the distillation teacher.
The human evaluation is particularly helpful. While the preference is slightly toward Dense H3, the large fraction of ties makes the claim that VDN largely preserves the quality of the original model considerably more convincing. The added VSA/SLA2 results also make the efficiency and quality comparison stronger.
My main remaining concern is the role of the reverse linear scan. The rebuttal explains the motivation for bidirectional memory, but its contribution is still not isolated experimentally. In particular, it remains unclear how much of the reported quality depends on the reverse pathway, while this choice also means that future latent chunks need to be available during inference.
Overall, the rebuttal addresses most of the issues I raised, and I feel more confident in my positive assessment of the paper.
Thank you for the encouraging assessment. Due to the time limit of the rebuttal period, we removed the reverse scan from the trained VDN-H3 checkpoint and reran inference without retraining.
As shown below, removing the reverse scan lowers VQA_T from 87.61 to 86.84 and also lowers FAST-VQA and MaxVQA clear-motion, while VQA_A and Q-Align remain essentially unchanged. The largest drops are in the temporal metrics, indicating that the trained model relies on the reverse pathway mainly for temporal quality.
| Method | VQA_A | VQA_T | Q-Align | FAST-VQA | MaxVQA clear-motion |
|---|---|---|---|---|---|
| VDN-H3 | 27.69 | 87.61 | 92.14 | 82.34 | 82.74 |
| VDN-H3 w/o reverse scan | 27.63 | 86.84 | 92.34 | 82.19 | 82.31 |
Regarding the availability of future latent chunks, the reverse scan adds no requirement beyond that of the base model. VDN-H3 replaces the full attention of Dense H3, which denoises all latent frames jointly and already lets each frame attend to future frames; the reverse scan gives the linear memory the same bidirectional context. Causal generation, by design, does not use future information, so in that setting we expect the reverse scan to be omitted and only the forward scan kept.
We will add this ablation and the discussion of causal generation to the revised paper. Thank you again for your support.
We are grateful to Reviewer nBo8 for the thoughtful assessment. Below, we respond to each weakness and question in turn; Question 1 is answered together with Weakness 1.
Weakness 1 & Question 1: Human evaluation of visual quality
We have added a method-blind, pairwise human preference study with 10 evaluators and 103 video pairs per model comparison. Model names were hidden; evaluators selected either video or a tie to record overall preference.
At 50 NFE, 59.2% of judgments are ties, suggesting that VDN-H3 largely preserves the perceptual quality of Dense H3. Together with the matched-resource speedups reported in Table 2, this supports the quality–efficiency trade-off of our attention design. At 8 NFE, evaluators favor VDN-H3 in 62.1% of judgments and FastH3 v2 in 25.2%, with 12.6% ties. This clear preference in the separate few-step comparison indicates that the advantages of the VDN-H3 system are perceptible to human viewers, supporting its practical value for efficient video generation.
Weakness 2: Baselines and related work
The submission already includes a training-based VSA baseline: FastH3 v2 uses VSA-H3 with 80% sparsity and DMD2 training. At the same 8 NFE, VDN-H3 achieves higher VQA_A, VQA_T, Q-Align, and FAST-VQA scores, as shown below. Its one-B200 denoising latency is also lower: 49.3 s versus 60.2 s for FastH3 v2.
We have additionally compared SLA2-H3 and VDN-H3 under the same training compute budget, using only a fraction of the compute required for full training. As shown below, VDN-H3 achieves higher scores on all four reported quality metrics: VQA_T improves from 75.91 to 86.66, and FAST-VQA from 77.44 to 81.74. This extends our evaluation to a trained sparse–linear baseline.
ARL² and ReHyAt share the principle of combining local Softmax attention with recurrent memory. VDN’s added value lies in three coordinated design choices. First, its frame-level joint update accounts for correlations among spatial tokens, reducing interference between their memory writes. Second, its attention pattern combines local cross-frame Softmax attention, bidirectional long-range memory, and global first/last-frame anchors to retain complementary temporal context. Third, normalization and branch-specific gates and output projections calibrate the Softmax and linear contributions, helping preserve quality when converting pretrained attention. We now discuss ARL² and ReHyAt in the Related Work section of the revised version (lines 509–516). All newly added or modified text in the revised paper is highlighted in red.