- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HetA-DiT to reduce attention computation in video diffusion models distilled with DMD. A lightweight branch predicts per-token uncertainty using a Gaussian likelihood objective. This uncertainty, combined with random token swapping, determines which tokens receive global attention and which receive local attention. During inference, uncertainty from the preceding denoising step is reused, so routing requires no additional Transformer evaluation. Experiments on Wan2.2 5B and Wan2.1 1.3B report generation quality competitive with three-step DMD and reductions in attention and Transformer block latency.
Critical strengths
-
The method addresses a relevant deployment setting. Reducing computation within each denoising step complements step distillation. Reusing the preceding step's uncertainty avoids an additional Transformer evaluation for routing during inference.
-
The evaluation includes practical measurements and relevant ablations. The experiments cover two model sizes, automatic quality metrics, a human preference study, measured block latency including amortized mask construction, and ablations of routing, local window size, and random selection. These results support the usefulness of heterogeneous attention in the tested settings.
Critical weaknesses
-
The overall quality and latency advantage over dense DMD is not established. Step distillation substantially reduces sampling cost relative to the original 50-step Wan models. The additional 1.63x speedup reported for Wan2.2 in Table 5, however, is measured at the Transformer block level, while the first denoising step remains dense. Quality also involves a tradeoff relative to three-step DMD. VBench 2.0 declines from 60.8 to 58.9 on Wan2.2, VBench declines from 83.61 to 83.10 on Wan2.1, and human preference rates are 32% for HetA-DiT versus 36% for three-step DMD. HetA-DiT is preferred over two-step DMD, with preference rates of 50% versus 24%. Whether it offers a better operating point than simply choosing the number of DMD steps therefore depends on end-to-end generation timing, which the paper does not report.
-
The independent contribution of the learned uncertainty signal is not isolated. In Table 6, the final method improves over fixed uniform routing by 0.15 points in total score and 0.94 points in semantic score. Table 8 shows that random swapping also matters. Uncertainty routing achieves a total score of 83.00 without random tokens and 83.27 with 7% random tokens. These results suggest that both uncertainty selection and random swapping affect performance, but leave unclear how much of the gain over fixed uniform routing comes from the learned signal when the swapping configuration is held fixed. A matched comparison using the same swapping configuration would clarify the independent value of the learned signal. Since the paper's central claim is that predicted uncertainty identifies where compute matters, this comparison is important for assessing the contribution.
Questions for authors
-
What are the end-to-end generation latencies of HetA-DiT compared with dense DMD using two and three steps under the same settings? Please include the first dense step and all routing costs. Does HetA-DiT offer a better tradeoff between quality and latency than either baseline?
-
Does fixed uniform routing in Table 6 use the same random swapping and training budget as the final method? If not, how does uncertainty routing compare with fixed uniform routing when both use 7% random swapping?
2 comments
Additional results: E2E latency improvements over various Wan models operating points
Following your question about generation latency, we have added decoder timing and output-fidelity measurements to complement our earlier response. These results provide a broader assessment of how HetA's attention and transformer-block savings carry through across model sizes, sampling schedules and decoder costs, beyond the original decoder-dominated configuration.
On the E2E latency improvements. The long original-decoder runtime comes largely from a latency-inefficient small-tile configuration. Keeping the original VAE weights and decoder implementation, changing only the tiling policy reduces measured 5B decoding from 30.88 s to 15.58 s with larger overlapping tiles, or 14.02 s untiled. This reduces the decoder bottleneck and makes savings from any DiT acceleration, including HetA, more visible in the total latency.
The previously measured 1.034x full-pipeline speedup for three-step 5B with current tiling is the least favorable original-decoder configuration below. The tables provide a broader view across common Wan model sizes and resolutions, with the commonly used three-step DMD and original 50-step denoising schedules. With untiled original decoding, the component-composed speedups reach 1.491x for three steps and 2.058x for the 100-NFE projection. They report HetA speedup for DiT plus decoding, using the same decoder choice and policy on both sides. These ratios combine separately measured components rather than directly timed full pipelines; text encoding and postprocessing are excluded.
3-step DMD
| Model and video size | Current tiling | Untiled | LightTAE |
|---|---|---|---|
| Wan2.1-1.3B, 81x480x832 | 1.069x | 1.103x | 1.183x |
| Wan2.2-5B, 121x704x1280 | 1.034x | 1.061x | 1.163x |
| Wan2.2-14B expert, 121x480x832 | 1.246x | 1.279x | 1.319x |
| Wan2.2-14B expert, 121x704x1280 | 1.455x | 1.491x | 1.537x |
50 steps + CFG (100 NFEs)
| Model and video size | Current tiling | Untiled | LightTAE |
|---|---|---|---|
| Wan2.1-1.3B, 81x480x832 | 1.291x | 1.300x | 1.308x |
| Wan2.2-5B, 121x704x1280 | 1.237x | 1.252x | 1.266x |
| Wan2.2-14B expert, 121x480x832 | 1.551x | 1.554x | 1.556x |
| Wan2.2-14B expert, 121x704x1280 | 2.055x | 2.058x | 2.062x |
All component measurements use A100-SXM4-80GB, BF16 and batch size one. Current tiling uses 256-pixel tiles with stride 192. The 14B-expert rows use synthetic-input implementation benchmarks. The 50-step CFG schedule projects 100 forward evaluations, including two dense first-step evaluations; quality at this schedule has not been evaluated. The broader tables show how the benefit grows when DiT computation dominates, particularly at larger model sizes and longer sampling schedules.
Empirical validation of LightTAE output fidelity. To validate the decoder substitution used in the latency estimates above, we decoded identical final HetA latents with LightTAE and the original untiled decoder on 20 fixed Wan2.2-5B prompts at 121x704x1280. LightTAE achieves mean PSNR 27.17 dB, SSIM 0.844 and LPIPS 0.0609 relative to the original decoder, supporting structural and perceptual similarity on this subset.
Response to tiqT: DMD latency comparison and matched routing control
We appreciate Reviewer tiqT's suggestions on generation time and fair routing comparisons. In summary:
Q1. How does HetA compare with two- and three-step DMD?
HetA is 1.17x faster than DMD3 over the complete DiT sequence (95% paired interval 1.16-1.17), including the first dense step and routing. The measured full pipeline is 1.034x faster because the current VAE decoder alone takes about 31.4 s.
This decoder bottleneck is complementary to attention acceleration. With the LightTAE decoder (about 0.21 s), keeping our measured non-decoder times unchanged projects about 9.82 s for DMD3 vs 8.46 s for HetA, or 1.16x full-pipeline speedup. We report this as a projection; the table reports the directly measured pipeline.
Against DMD2, HetA improves both benchmarks and wins 50% of human comparisons vs 24%, with 26% ties. Against DMD3, human preference is roughly balanced (32% HetA, 32% ties, 36% DMD3): HetA is faster and scores higher on VBench, while DMD3 scores higher on VBench2.0. HetA therefore occupies a different point on the speed-quality frontier rather than simply reproducing a lower DMD step count.
The scaling trend is stronger for larger models: trained Wan2.1-14B gives 1.18x complete-DiT speedup at three steps (26.34 vs 30.93 s). Separately, a Wan2.2-14B expert forward at 121x704x1280 takes 66.63 s dense vs 31.61 s with HetA (2.11x); this implementation-only test uses synthetic inputs and no trained uncertainty head.
Q2. Did the submitted uniform baseline use the same swapping and training budget? No. The submitted uniform baseline used no random swaps and a larger 13x13x13 window. To provide the comparison you requested, we trained a full-budget uniform model with the same 11x11x11 neighborhood and 7% swapping as HetA.
*The no-swap uniform result uses a larger 13x13x13 window; at that window, HetA scores 83.36 (Table 12).
With the same swaps, neighborhood and adaptation budget, uncertainty routing scores 83.28 vs 82.68, a +0.60-point gain (95% paired prompt interval +0.20 to +0.97), on all 16 dimensions and 944 prompts with five samples each.
As a complementary test, the same matched HetA and uniform models score 62.14 vs 60.42 (+1.72) on a 128-prompt VBench2.0 subset. The smaller result is not statistically significant but is directionally consistent with the significant full-VBench comparison.
Together, these results separate the routing effect from random swapping and show that HetA's efficiency-quality tradeoff is not reproduced simply by reducing the number of DMD steps.