acceptodds
Withdrawn from ICLR 2027

Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Efficient video generation requires reducing the quadratic cost of self-attention over long spatio-temporal token sequences. Existing efficient-attention methods typically apply the same computation pattern to every token, even though denoising difficulty varies substantially across video regions and evolves throughout the generation process. We introduce HetA-DiT, a heterogeneous attention mechanism that adaptively allocates computation according to token difficulty. A lightweight uncertainty branch predicts a token-wise estimate of denoising difficulty, which is used to route uncertain tokens through dense global attention while processing more reliable tokens with efficient local attention. The resulting routing is content- and timestep-adaptive, retains global context where it matters most, and provides a single parameter for controlling the quality-efficiency trade-off. HetA-DiT is compatible with few-step distribution-matching distillation and introduces no additional Transformer evaluation at inference time by reusing uncertainty estimates from the preceding denoising step. We evaluate the method on DMD-distilled Wan2.2-5B and Wan2.1-1.3B models. Across VBench, VBench-2.0, and human preference evaluation, HetA-DiT maintains competitive generation quality while routing only approximately 20% of tokens through dense attention.

Primary area:
generative models
Keywords:
Efficient Video Generation, Video Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_tiqTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HetA-DiT to reduce attention computation in video diffusion models distilled with DMD. A lightweight branch predicts per-token uncertainty using a Gaussian likelihood objective. This uncertainty, combined with random token swapping, determines which tokens receive global attention and which receive local attention. During inference, uncertainty from the preceding denoising step is reused, so routing requires no additional Transformer evaluation. Experiments on Wan2.2 5B and Wan2.1 1.3B report generation quality competitive with three-step DMD and reductions in attention and Transformer block latency.

Critical strengths
  1. The method addresses a relevant deployment setting. Reducing computation within each denoising step complements step distillation. Reusing the preceding step's uncertainty avoids an additional Transformer evaluation for routing during inference.

  2. The evaluation includes practical measurements and relevant ablations. The experiments cover two model sizes, automatic quality metrics, a human preference study, measured block latency including amortized mask construction, and ablations of routing, local window size, and random selection. These results support the usefulness of heterogeneous attention in the tested settings.

Critical weaknesses
  1. The overall quality and latency advantage over dense DMD is not established. Step distillation substantially reduces sampling cost relative to the original 50-step Wan models. The additional 1.63x speedup reported for Wan2.2 in Table 5, however, is measured at the Transformer block level, while the first denoising step remains dense. Quality also involves a tradeoff relative to three-step DMD. VBench 2.0 declines from 60.8 to 58.9 on Wan2.2, VBench declines from 83.61 to 83.10 on Wan2.1, and human preference rates are 32% for HetA-DiT versus 36% for three-step DMD. HetA-DiT is preferred over two-step DMD, with preference rates of 50% versus 24%. Whether it offers a better operating point than simply choosing the number of DMD steps therefore depends on end-to-end generation timing, which the paper does not report.

  2. The independent contribution of the learned uncertainty signal is not isolated. In Table 6, the final method improves over fixed uniform routing by 0.15 points in total score and 0.94 points in semantic score. Table 8 shows that random swapping also matters. Uncertainty routing achieves a total score of 83.00 without random tokens and 83.27 with 7% random tokens. These results suggest that both uncertainty selection and random swapping affect performance, but leave unclear how much of the gain over fixed uniform routing comes from the learned signal when the swapping configuration is held fixed. A matched comparison using the same swapping configuration would clarify the independent value of the learned signal. Since the paper's central claim is that predicted uncertainty identifies where compute matters, this comparison is important for assessing the contribution.

Questions for authors
  1. What are the end-to-end generation latencies of HetA-DiT compared with dense DMD using two and three steps under the same settings? Please include the first dense step and all routing costs. Does HetA-DiT offer a better tradeoff between quality and latency than either baseline?

  2. Does fixed uniform routing in Table 6 use the same random swapping and training budget as the final method? If not, how does uncertainty routing compare with fixed uniform routing when both use 7% random swapping?

2 comments
AuthorsOpenReview ↗

Response to tiqT: DMD latency comparison and matched routing control

We appreciate Reviewer tiqT's suggestions on generation time and fair routing comparisons. In summary:

  • Faster than DMD3: HetA gives 1.17x complete-DiT speedup, including the first dense step and routing, with higher VBench (83.28 vs 82.94).
  • Higher quality than DMD2: HetA improves both benchmarks and wins 50% of human comparisons vs 24% for DMD2, with 26% ties.
  • Matched uniform control: uncertainty routing improves full VBench by +0.60 points (95% paired prompt interval +0.20 to +0.97).

Q1. How does HetA compare with two- and three-step DMD?

MethodDiT median (s)Full pipeline (s)VBenchVBench2.0Human: dense / tie / HetA
Dense DMD26.3437.8382.4857.9524% / 26% / 50%
Dense DMD39.5241.0182.9460.8036% / 32% / 32%
HetA38.1539.6583.2858.92-

HetA is 1.17x faster than DMD3 over the complete DiT sequence (95% paired interval 1.16-1.17), including the first dense step and routing. The measured full pipeline is 1.034x faster because the current VAE decoder alone takes about 31.4 s.

This decoder bottleneck is complementary to attention acceleration. With the LightTAE decoder (about 0.21 s), keeping our measured non-decoder times unchanged projects about 9.82 s for DMD3 vs 8.46 s for HetA, or 1.16x full-pipeline speedup. We report this as a projection; the table reports the directly measured pipeline.

Against DMD2, HetA improves both benchmarks and wins 50% of human comparisons vs 24%, with 26% ties. Against DMD3, human preference is roughly balanced (32% HetA, 32% ties, 36% DMD3): HetA is faster and scores higher on VBench, while DMD3 scores higher on VBench2.0. HetA therefore occupies a different point on the speed-quality frontier rather than simply reproducing a lower DMD step count.

The scaling trend is stronger for larger models: trained Wan2.1-14B gives 1.18x complete-DiT speedup at three steps (26.34 vs 30.93 s). Separately, a Wan2.2-14B expert forward at 121x704x1280 takes 66.63 s dense vs 31.61 s with HetA (2.11x); this implementation-only test uses synthetic inputs and no trained uncertainty head.

Q2. Did the submitted uniform baseline use the same swapping and training budget? No. The submitted uniform baseline used no random swaps and a larger 13x13x13 window. To provide the comparison you requested, we trained a full-budget uniform model with the same 11x11x11 neighborhood and 7% swapping as HetA.

RoutingNo swaps7% swaps (matched)
Uniform83.13*82.68
Uncertainty83.0083.28

*The no-swap uniform result uses a larger 13x13x13 window; at that window, HetA scores 83.36 (Table 12).

With the same swaps, neighborhood and adaptation budget, uncertainty routing scores 83.28 vs 82.68, a +0.60-point gain (95% paired prompt interval +0.20 to +0.97), on all 16 dimensions and 944 prompts with five samples each.

As a complementary test, the same matched HetA and uniform models score 62.14 vs 60.42 (+1.72) on a 128-prompt VBench2.0 subset. The smaller result is not statistically significant but is directionally consistent with the significant full-VBench comparison.

Together, these results separate the routing effect from random swapping and show that HetA's efficiency-quality tradeoff is not reproduced simply by reducing the number of DMD steps.

AuthorsOpenReview ↗

Additional results: E2E latency improvements over various Wan models operating points

Following your question about generation latency, we have added decoder timing and output-fidelity measurements to complement our earlier response. These results provide a broader assessment of how HetA's attention and transformer-block savings carry through across model sizes, sampling schedules and decoder costs, beyond the original decoder-dominated configuration.

On the E2E latency improvements. The long original-decoder runtime comes largely from a latency-inefficient small-tile configuration. Keeping the original VAE weights and decoder implementation, changing only the tiling policy reduces measured 5B decoding from 30.88 s to 15.58 s with larger overlapping tiles, or 14.02 s untiled. This reduces the decoder bottleneck and makes savings from any DiT acceleration, including HetA, more visible in the total latency.

The previously measured 1.034x full-pipeline speedup for three-step 5B with current tiling is the least favorable original-decoder configuration below. The tables provide a broader view across common Wan model sizes and resolutions, with the commonly used three-step DMD and original 50-step denoising schedules. With untiled original decoding, the component-composed speedups reach 1.491x for three steps and 2.058x for the 100-NFE projection. They report HetA speedup for DiT plus decoding, using the same decoder choice and policy on both sides. These ratios combine separately measured components rather than directly timed full pipelines; text encoding and postprocessing are excluded.

3-step DMD

Model and video sizeCurrent tilingUntiledLightTAE
Wan2.1-1.3B, 81x480x8321.069x1.103x1.183x
Wan2.2-5B, 121x704x12801.034x1.061x1.163x
Wan2.2-14B expert, 121x480x8321.246x1.279x1.319x
Wan2.2-14B expert, 121x704x12801.455x1.491x1.537x

50 steps + CFG (100 NFEs)

Model and video sizeCurrent tilingUntiledLightTAE
Wan2.1-1.3B, 81x480x8321.291x1.300x1.308x
Wan2.2-5B, 121x704x12801.237x1.252x1.266x
Wan2.2-14B expert, 121x480x8321.551x1.554x1.556x
Wan2.2-14B expert, 121x704x12802.055x2.058x2.062x

All component measurements use A100-SXM4-80GB, BF16 and batch size one. Current tiling uses 256-pixel tiles with stride 192. The 14B-expert rows use synthetic-input implementation benchmarks. The 50-step CFG schedule projects 100 forward evaluations, including two dense first-step evaluations; quality at this schedule has not been evaluated. The broader tables show how the benefit grows when DiT computation dominates, particularly at larger model sizes and longer sampling schedules.

Empirical validation of LightTAE output fidelity. To validate the decoder substitution used in the latency estimates above, we decoded identical final HetA latents with LightTAE and the original untiled decoder on 20 fixed Wan2.2-5B prompts at 121x704x1280. LightTAE achieves mean PSNR 27.17 dB, SSIM 0.844 and LPIPS 0.0609 relative to the original decoder, supporting structural and perceptual similarity on this subset.

Reviewer_gqn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HetA-DiT, a heterogeneous attention mechanism for efficient video diffusion. The key idea is to allocate attention computation according to token-level denoising difficulty, using dense global attention for uncertain tokens and efficient local attention for more reliable ones. The authors introduce an uncertainty prediction branch trained with a modified DMD objective and use its predictions to guide adaptive token routing. Experiments on DMD-distilled Wan2.2-5B and Wan2.1-1.3B demonstrate reduced attention computation and improved Transformer efficiency while maintaining competitive video generation quality.

Critical strengths
  1. The paper addresses an important computational bottleneck in video diffusion by allocating attention computation non-uniformly across tokens. The proposed approach is compatible with few-step DMD-distilled models and avoids additional Transformer evaluations during inference. The reported Transformer block speedups demonstrate its practical potential.

  2. The authors evaluate HetA-DiT on two Wan model variants using VBench, VBench-2.0, and human preference studies. Comparisons with existing efficient attention methods and ablations of routing strategies provide evidence that the approach can reduce computational costs while maintaining competitive generation quality.

Critical weaknesses
  1. Insufficient justification for using denoising uncertainty as a proxy for global attention requirements. The central assumption is that tokens with higher denoising uncertainty benefit more from dense global attention. However, the uncertainty branch is trained using reconstruction-based supervision rather than directly measuring the benefit of global versus local attention. High denoising uncertainty does not necessarily imply that global attention is required. While the routing ablations provide indirect support, the paper lacks a direct analysis establishing the relationship between predicted uncertainty and the actual benefit of dense attention. This weakens the justification for the core routing criterion.

  2. Limited improvements over simpler routing strategies and incomplete efficiency–quality evaluation. The proposed uncertainty-guided routing achieves only modest improvements over fixed uniform routing. For example, Table 6 reports VBench scores of 83.28 for HetA-DiT and 83.13 for fixed routing, suggesting that much of the benefit may come from heterogeneous attention itself rather than the proposed uncertainty-based allocation. Furthermore, although HetA-DiT preserves competitive VBench performance, its VBench-2.0 score decreases from 60.8 for the 3-step DMD baseline to 58.9. The reported 1.63× speedup on Wan2.2-5B is measured at the Transformer block level rather than for the complete video generation pipeline. These observations make it difficult to fully assess whether the additional uncertainty prediction and training complexity are justified by the resulting quality–efficiency improvements.

Questions for authors

Q1. Validity of uncertainty-guided routing. Could the authors provide a direct analysis showing that tokens with higher predicted uncertainty benefit more from dense global attention? For example, measuring the correlation between uncertainty and the difference between dense- and local-attention outputs would help validate the central assumption.

Q2. Benefits over simpler routing strategies. Given the relatively small VBench improvement over fixed uniform routing (83.28 vs. 83.13), could the authors further clarify the independent contribution of uncertainty-guided routing? Are these improvements statistically significant across different prompts and random seeds?

Q3. Quality–efficiency trade-off. HetA-DiT achieves a lower VBench-2.0 score than the 3-step DMD baseline (58.9 vs. 60.8). Could the authors discuss which quality dimensions are most affected by heterogeneous attention and whether the degradation increases at higher sparsity levels?

Q4. End-to-end efficiency. The paper reports substantial attention-level and Transformer-block speedups. Could the authors additionally report end-to-end video generation latency, including all inference components, to clarify the practical acceleration achieved by HetA-DiT?

I find the research direction meaningful and the efficiency results encouraging. I am open to raising my score if the authors can adequately address these concerns during the rebuttal.

5 comments
AuthorsOpenReview ↗

Response to gqn3: Uncertainty routing, quality tradeoffs, and generation latency

We thank Reviewer gqn3 for the thoughtful feedback on uncertainty and video quality. In summary, our new results show:

  • Matched routing: HetA improves full VBench by +0.60 over uniform routing (95% paired prompt interval +0.20 to +0.97).
  • Meaningful routing signal: on disjoint prompts, uncertainty-selected tokens capture 1.29x the output change expected from random selection.
  • Faster denoising: HetA is 1.17x faster than dense DMD3 over the complete DiT sequence while scoring higher on VBench (83.28 vs 82.94).

Q1. Do uncertain tokens depend more on global attention? Yes. Higher uncertainty tends to identify tokens whose predictions change more when global context is removed.

We compare preceding-step uncertainty with the final-denoiser change between dense and local attention under matched inputs:

Prompt setMean Spearman correlation95% interval
Initial eight prompts0.319[0.227, 0.415]
Eight disjoint prompts0.359[0.289, 0.439]

On the two cohorts, uncertainty selects 17.40% and 18.25% of eligible tokens before swaps but captures 21.94% and 23.60% of the squared prediction change. This is 1.26x random expectation (95% interval 1.20-1.33) and 1.29x (1.21-1.38), respectively. Intervals resample prompts while keeping seeds and denoising steps together.

The correlation at the individual attention-module output is weak (0.030 [-0.033, 0.107]), while the whole-denoiser output shows the reproducible 0.32-0.36 relationship above. Thus, the signal is more predictive of downstream denoising sensitivity than of an individual attention layer.

Q2. Does uncertainty improve on uniform routing? Yes. With the same 11x11x11 neighborhood, 7% swaps and adaptation budget, uncertainty routing scores 83.28 vs 82.68, a +0.60-point gain (95% paired prompt interval +0.20 to +0.97).

RoutingNo swaps7% swaps (matched)
Uniform83.13*82.68
Uncertainty83.0083.28

*The no-swap uniform result uses a larger 13x13x13 window; at that window, HetA scores 83.36 (Table 12).

The interval uses 20,000 paired prompt bootstraps, keeping each prompt's five samples together and recomputing VBench. Local-only attention scores 82.85 total but 79.99 semantic; random and uniform routing add at most 0.59 semantic points, while HetA reaches 81.52 (+1.53).

As a complementary test, the same matched HetA and uniform models score 62.14 vs 60.42 (+1.72) on a 128-prompt VBench2.0 subset. This smaller result is not statistically significant but is directionally consistent with the full-VBench comparison.

Q3. What happens to quality at higher sparsity?

Local-attention fractionVBench qualityVBench semanticVBench total
70%83.7381.3083.24
80%83.7181.5283.28
90%83.4681.1182.99

VBench is stable from 70% to 80% local attention; at 90% it drops by 0.29 points, about 0.27 of which comes from Dynamic Degree. At 80%, VBench2.0 is 58.92 vs 60.80 for dense DMD3; the largest losses are in Dynamic Attribute, Motion Rationality and Diversity, with gains in Instance Preservation, Motion Order Understanding and Complex Landscape.

Q4. How much time does HetA save over a full generation?

MethodDiT (s)Full pipeline (s)VBenchVBench2.0Human: dense / tie / HetA
Dense DMD26.3437.8382.4857.9524% / 26% / 50%
Dense DMD39.5241.0182.9460.8036% / 32% / 32%
HetA38.1539.6583.2858.92-

HetA is 1.17x faster than DMD3 over the complete DiT sequence (95% paired interval 1.16-1.17), including the dense first step and routing overheads. The measured full-pipeline gain is 1.034x because the current VAE decoder alone takes about 31.4 s. This decoder bottleneck is complementary to HetA. With an efficient decoer like LightTAE decoder (about 0.21 s), keeping our measured non-decoder times unchanged projects full-pipeline times of about 9.82 s for DMD3 and 8.46 s for HetA, or approximately 1.16x end-to-end speedup. We report this only as a projection; the measured result remains 41.01 vs 39.65 s.

Against DMD2, HetA improves both benchmarks and wins 50% of human comparisons vs 24%, with 26% ties. Against DMD3, human preference is roughly balanced (32% HetA, 32% ties, 36% DMD3), while HetA is faster and scores higher on VBench but lower on VBench2.0.

I appreciate the authors' detailed responses and additional experiments. The new uncertainty-sensitivity analysis and matched routing comparisons provide stronger evidence for the effectiveness of uncertainty-guided routing.

However, I remain concerned about the overall quality–efficiency trade-off. While HetA achieves a 1.17× speedup over the complete DiT sequence, the measured end-to-end speedup is only 1.034×. Although the authors suggest that using LightTAE could improve the end-to-end speedup to approximately 1.16×, this remains a projection rather than an experimentally validated result.

Moreover, despite a slight improvement in VBench, VBench-2.0 decreases from 60.80 to 58.92 with some substantial degradations, for example, Dynamic Attribute (−10.26), Motion Rationality (−9.20), and Diversity (−6.88). While the authors identify these degraded dimensions, the reasons for this discrepancy remain insufficiently explained.

Overall, although the rebuttal addresses several of my original concerns, I am not fully convinced that the reported acceleration justifies the observed quality trade-offs. I therefore incline to maintain my initial rating.

We thank the reviewer for acknowledging that the new uncertainty-sensitivity analysis and matched routing comparisons strengthen the evidence for uncertainty-guided routing. As you note, HetA delivers substantial self-attention speedups, the component it targets, alongside transformer-block speedups. Your observation motivates a broader assessment of how these savings carry through across model sizes, sampling schedules and decoder costs, as a single decoder-dominated setup captures only part of HetA's practical benefits and the broader value of attention efficiency.

On the E2E latency improvements. Following your concern about the projected LightTAE result, we added decoder latency and output-fidelity measurements. The long original-decoder runtime comes largely from a latency-inefficient small-tile configuration. Keeping the original VAE weights and decoder implementation, changing only the tiling policy reduces measured 5B decoding from 30.88 s to 15.58 s with larger overlapping tiles, or 14.02 s untiled. This reduces the decoder bottleneck and makes savings from any DiT acceleration, including HetA, more visible in the total latency.

The previously measured 1.034x full-pipeline speedup for three-step 5B with current tiling is the least favorable original-decoder configuration below. The tables provide a broader view across common Wan model sizes and resolutions, with the commonly used three-step DMD and original 50-step denoising schedules. With untiled original decoding, the component-composed speedups reach 1.491x for three steps and 2.058x for the 100-NFE projection. They report HetA speedup for DiT plus decoding, using the same decoder choice and policy on both sides. These ratios combine separately measured components rather than directly timed full pipelines; text encoding and postprocessing are excluded.

3-step DMD

Model and video sizeCurrent tilingUntiledLightTAE
Wan2.1-1.3B, 81x480x8321.069x1.103x1.183x
Wan2.2-5B, 121x704x12801.034x1.061x1.163x
Wan2.2-14B expert, 121x480x8321.246x1.279x1.319x
Wan2.2-14B expert, 121x704x12801.455x1.491x1.537x

50 steps + CFG (100 NFEs)

Model and video sizeCurrent tilingUntiledLightTAE
Wan2.1-1.3B, 81x480x8321.291x1.300x1.308x
Wan2.2-5B, 121x704x12801.237x1.252x1.266x
Wan2.2-14B expert, 121x480x8321.551x1.554x1.556x
Wan2.2-14B expert, 121x704x12802.055x2.058x2.062x

All component measurements use A100-SXM4-80GB, BF16 and batch size one. Current tiling uses 256-pixel tiles with stride 192. The 14B-expert rows use synthetic-input implementation benchmarks. The 50-step CFG schedule projects 100 forward evaluations, including two dense first-step evaluations; quality at this schedule has not been evaluated. The broader tables show how the benefit grows when DiT computation dominates, particularly at larger model sizes and longer sampling schedules.

Empirical validation of LightTAE output fidelity. To validate the decoder substitution used in the latency estimates above, we decoded identical final HetA latents with LightTAE and the original untiled decoder on 20 fixed Wan2.2-5B prompts at 121x704x1280. LightTAE achieves mean PSNR 27.17 dB, SSIM 0.844 and LPIPS 0.0609 relative to the original decoder, supporting structural and perceptual similarity on this subset.

Reviewer_KGpfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HetA-DiT, which aims to reduce the computational cost of self-attention in few-step video diffusion models. Through two-stage training, the method first learns token-wise uncertainty using an improved DMD objective, and then uses the predicted scores to assign more difficult positions to global attention while applying local attention to the remaining positions. During inference, the method reuses uncertainty from the previous denoising step to guide routing at the next step, avoiding an additional full-model forward pass.

Critical strengths
  1. The method presents a complete design that accounts for routing overhead. The paper integrates uncertainty prediction, global/local attention allocation, and DMD fine-tuning into a complete pipeline. By reusing predictions from the previous denoising step, the method avoids adding a full-model forward pass for routing; the local path also continuously updates the current token features.

  2. Multiple quality evaluations and actual latency measurements are provided. The experiments combine VBench, VBench-2.0, human preference evaluation, and component ablations, and use FlexAttention to verify that theoretical computational savings can translate into actual speed gains.

Critical weaknesses
  1. The methodological novelty is relatively limited. AHA (Learning When Not to Attend Globally) has already proposed the core allocation paradigm of using learnable scores to select global or local attention on a per-token basis. In comparison, the new contributions of HetA-DiT mainly lie in uncertainty supervision and routing scheduling across denoising steps for few-step video diffusion.

  2. The benefits of uncertainty-based routing are not yet sufficiently demonstrated. Table 6 shows that, compared with fixed uniform routing, uncertainty-based routing provides limited overall gains, which are primarily concentrated in semantic metrics.

  3. The experimental scale and coverage are limited. Quality evaluation mainly focuses on few-step distilled models in the Wan family with relatively limited parameter counts, without validation on larger models, other model families, or conventional multi-step generation settings.

Questions for authors
  1. Please clarify the substantive differences from AHA and why they are necessary. Beyond the difference in application scenarios, what specific problems with directly learning gating decisions are addressed by uncertainty supervision and scheduling across denoising steps, respectively?

  2. Please isolate the contributions of uncertainty-based selection and random swapping. In Table 8, the total score of uncertainty-based routing with random swapping disabled is 83.00, lower than the 83.13 achieved by fixed uniform routing in Table 6. Please compare fixed uniform routing and uncertainty-based routing, each with random swapping enabled and disabled, under the same training settings and budget.

  3. Please provide additional evidence of generalization. Add a representative quality evaluation experiment on a larger model or a model from another family, and explain the method's sensitivity to different numbers of denoising steps.

3 comments
AuthorsOpenReview ↗

Response to KGpf (1/2): AHA comparison

We thank Reviewer KGpf for the thoughtful comments on AHA, routing and model scale. In summary, our new results show:

  • AHA comparison: HetA improves full VBench by +0.72 points (95% paired prompt interval +0.36 to +1.08), with higher quality and semantic scores, while selecting slightly fewer global queries (about 17-18% vs AHA's 20%).
  • Shared routes save work: HetA is 1.22x faster than a separately optimized AHA in preliminary complete-DiT timing at equal global-query counts.
  • Matched routing control: uncertainty routing beats uniform routing by +0.60 points (95% interval +0.20 to +0.97).
  • Larger scale: trained Wan2.1-14B HetA runs 1.18x faster over the complete DiT; quality is still converging after a short adaptation.

We address each point below (part 1: AHA; part 2: routing and scale). Changes are in the main text and appendix, highlighted in blue; the HetA kernel is described in our response to istB.

Q1. How does HetA compare with AHA? HetA scores higher on both quality and semantics at similar sparsity: 83.28 versus 82.56 total, +0.72 points (95% paired prompt interval +0.36 to +1.08).

MethodVBench qualityVBench semanticVBench total
HetA83.7181.5283.28
AHA83.0280.7082.56

AHA adaptation. We implemented AHA on Wan2.2-5B, keeping its token/head gates, straight-through estimator, task loss and sparsity loss, with a 3D video neighborhood and DMD training. Both methods use the same dense initialization, neighborhood, adaptation budget, dense first step and 7% swaps, with three steps at 121x704x1280 and the full VBench suite (944 prompts, five samples each, all 16 dimensions). AHA keeps its top-scoring 20% of eligible token/head positions.

Design distinction. AHA predicts token/head routes from current-layer features, trained through task and sparsity losses. HetA explicitly supervises uncertainty with DMD residuals: we train the head with dense attention, then freeze it during HetA adaptation. Routing uses the previous step's uncertainty, so it needs no extra transformer evaluation; the first step is dense. To our knowledge, HetA is the first use of uncertainty-supervised tokenwise global/local attention in few-step video generation.

Runtime against an optimized AHA. We optimized AHA's head-specific routing and packing with HetA's local 3D Triton kernel and FlashAttention for global queries, testing both padding all heads and grouping heads in fours. Both pass independent reference checks. At exactly equal global-query counts, preliminary three-step 5B timing gives 8.13 s for HetA versus 9.98 s for AHA (1.22x), over two paired prompt/seed cases with an intermediate AHA checkpoint, including the dense first step, routing, swaps and packing.

Attention / complete-block speedup over dense FlashAttention (nine interleaved repeats):

Model / resolutionHetAFaster AHA variant
Wan2.1-1.3B / 480x8321.61x / 1.27x1.04x / 1.03x
Wan2.2-5B / 704x12801.90x / 1.28x1.00x / 1.00x
Wan2.2-14B expert / 480x8322.72x / 1.57x1.86x / 1.36x
Wan2.2-14B expert / 704x12803.75x / 2.22x3.27x / 2.08x

At 5B, AHA's gate evaluation, per-head routing and packing cancel its savings. HetA reuses one uncertainty map across blocks and shares token selection across heads. AHA needs a separate query list per head: padding to the largest head uses a median 2.51x the selected global slots, and grouping four heads reduces this to 1.17x but needs six global FlashAttention calls per block, versus one for HetA. Only the 5B row uses trained AHA gates; other sizes use seeded routers to test implementation scaling.

The uncertainty diagnostics (1.26x and 1.29x random capture, replicated) are in our response to gqn3 and Appendix Table 14.

AuthorsOpenReview ↗

Response to KGpf (2/2): routing control and larger models

(Part 2 of 2)

Q2. Routing choices with the same random swaps. With the same neighborhood, 7% swaps and adaptation budget, uncertainty routing scores 83.28 versus 82.68 for uniform: +0.60 points (95% paired prompt interval +0.20 to +0.97).

RoutingNo swaps7% swaps (matched)
Uniform83.13*82.68
Uncertainty83.0083.28

*Uses a larger 13x13x13 window; at that window, HetA scores 83.36 (Table 12).

Local-only attention reaches 82.85 total but 79.99 semantic. Across Table 6, random and uniform routing add at most 0.59 semantic points, while HetA with swaps reaches 81.52 (+1.53): targeted global attention matters most for semantics.

Q3. Larger models and denoising step count. Following your suggestion, we added three-step Wan2.1-14B results at 81x480x832. HetA gives 1.18x complete-DiT speedup (30.93 s vs 26.34 s; 95% paired interval 1.17-1.18), including the first dense step, uncertainty prediction and routing.

Quality is still converging. Retraining at 14B is compute-intensive, so within the rebuttal period we could run only a short schedule: the dense LoRA-based student needed about 30k DMD iterations, while HetA adapts the same attention LoRA parameters with only 1,625 generator updates, after an uncertainty stage with about 7% of the 5B schedule:

Training stageSubmitted 5BPreliminary 14B
Uncertainty training, dense attention7,500 generator updates500 updates, uncertainty heads only
HetA adaptation, uncertainty frozen3,000 generator updates1,625 generator updates
Wan2.1-14B, 3 steps, 480pDiT median (s)VBench qualityVBench semanticVBench total
Dense DMD330.9385.2981.9784.63
HetA3, 1,625 updates26.3483.0880.0282.47

At this checkpoint HetA's total is 2.16 points below dense, and we expect the gap to narrow with longer training; we are extending both stages and will report the final results in the paper. Even after this short schedule, Appendix Figures 12-13 show prompt-matched samples with recognizable subjects, coherent layouts and the requested style, with differences mainly in saturation and framing. Details are in Appendix B.2.

HetA only needs standard DiT self-attention over spatiotemporal tokens; our experiments cover Wan2.1-1.3B, Wan2.2-5B and Wan2.1-14B, and another model family remains future work.

On step count: a pretrained Wan2.2-14B expert forward at 121x704x1280 takes 66.63 s dense versus 31.61 s with HetA (2.11x; synthetic inputs, no trained uncertainty head). Including one dense first step, this projects 1.54x DiT speedup at three steps and 2.06x at 50, since later steps amortize the dense first step. With more steps, consecutive steps are also closer, so the previous-step uncertainty should remain a good routing signal; we have not yet measured quality at other step counts.

AuthorsOpenReview ↗

Update on Q3: longer 14B adaptation narrows the quality gap

Following up on Q3 (larger models): we continued the Wan2.1-14B HetA adaptation from 1,625 to 3,250 generator updates, with the same routing settings and the uncertainty head still frozen after its 500 updates. On the same full VBench protocol (944 prompts, five samples each, all 16 dimensions):

Wan2.1-14B, 3 steps, 480pVBench qualityVBench semanticVBench total
Dense DMD385.2981.9784.63
HetA3, 1,625 updates83.0880.0282.47
HetA3, 3,250 updates83.9180.6383.25

Doubling the adaptation budget narrows the gap to dense from 2.16 to 1.38 points, improving both quality (+0.83) and semantic (+0.61) scores. This supports the short rebuttal-period schedule as the main cause of the remaining gap. Training continues, and we will report the final results in the paper.

Reviewer_istBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HetA-DiT, which reuses uncertainty estimates from the previous denoising step to route difficult video tokens to global attention and easier tokens to local attention, reducing computation while maintaining competitive generation quality in few-step video diffusion.

Critical strengths
  1. The idea is interesting
  2. The paper is well written
Critical weaknesses

I just have one major question.

Whether the proposed token-level sparsity translates into hardware-efficient computation. Under a conventional block-sparse layout, a query tile containing even one globally attending query must access all key/value tiles, potentially eliminating block-level sparsity for that query tile despite most of its queries requiring only local attention. This issue is particularly relevant when global queries are spatially dispersed or introduced through the proposed per-Transformer-block random swapping. Consequently, the theoretical reduction in token-level FLOPs does not necessarily imply a comparable reduction in executed GPU operations. Dynamic mask construction, irregular memory accesses, and any query packing or output scattering may further offset the computational savings.

Although Table 5 reports measured attention and Transformer-block speedups of 1.84× and 1.63×, respectively, and states that amortized BlockMask construction costs are included, the implementation and timing protocol remain insufficiently specified. The authors should clarify the query/KV tile sizes, whether queries are regrouped by attention type, and how mask construction and updates are amortized when routing changes across Transformer blocks. They should also report the actual block-level sparsity under uncertainty-guided routing with random swapping, provide a latency breakdown covering routing and mask-management overheads, and specify the dense baseline implementation used for comparison. These details are necessary to understand how the reported acceleration is achieved and to assess its reproducibility; simply identifying FlexAttention as the backend does not resolve the concern.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Separate local/global kernels: 3.78× attention speedup, routing costs included

We are grateful to Reviewer istB for the careful questions about GPU execution. In summary, our new results show:

  • Token sparsity becomes GPU savings: local and global queries run in separate calls; in one mixed layout, about 99.99% of query tiles would load all keys.
  • Measured speedups: against dense FlashAttention, HetA reaches 3.78x attention and 2.23x block speedup at 14B/704p.
  • Routing included: the complete 5B DiT sequence runs 1.17x faster than dense DMD3, including the first dense step, uncertainty prediction and all routing costs.

We address each point below; implementation details and updated measurements are in the main text and appendix, highlighted in blue.

To address your concern about GPU efficiency, we built an optimized Triton kernel for HetA's existing attention rule. In the same A100 attention benchmark, it takes 22.19 ms compared with 44.82 ms for FlexAttention, approximately 2.02x faster. We use this kernel for the new timings below.

How the kernels handle routes and swaps. Before each sparse block, we randomly exchange equal-sized subsets of local and global queries (up to 7% of token positions from each group). Group sizes and token positions stay the same; only attention assignments change.

After swapping, local queries go to the Triton kernel, which uses their clipped 3D neighborhoods, and global queries go to a separate FlashAttention call over all keys and values. Outputs are scattered back to their original positions. Separating the calls keeps a global query from forcing a local query tile to load all keys and values.

For Wan2.2-5B, the local kernel uses 16 query rows in 2x2x4 spatiotemporal tiles and chunks of 64 keys. We cache the neighborhood geometry and rebuild the query assignments after each block's swaps. The timings include rebuilding, gathering and scattering, and the generation benchmark also includes uncertainty prediction and selection.

Why separate calls matter. To investigate the tile mixing you highlighted, we examined the saved routes from an uncertainty-guided generation with random swaps. Across its 60 sparse blocks, approximately 74.35% of queries are local. Mixing local and global queries would make 99.9961% of flat 64-query tiles load all key/value tiles; even spatial 16-query tiles reach 91.88%. Separate calls avoid this loss of sparsity. Inactive rows, clipped neighborhoods and padding still give our local layout 2.863x as many tile slots as the ideal local query/key count; the measured times include this padding and both attention paths.

Comparison with dense FlashAttention. We use PyTorch FlashAttention with the same BF16 inputs and layouts. A tuned dense Triton baseline was slower on all four workloads.

ModelFrames x height x widthAttention speedupTransformer-block speedup
Wan2.1-1.3B81x480x8321.75x1.32x
Wan2.2-5B121x704x12802.05x1.30x
Wan2.2-14B expert121x480x8322.77x1.57x
Wan2.2-14B expert121x704x12803.78x2.23x

We used A100-SXM4-80GB, PyTorch 2.8.0+cu128, BF16, batch size one, synthetic inputs, 80% local queries and new 7% swaps at each block, reporting synchronized medians over 15 repeats after warmup, including route construction, gathering and scattering. The submitted Table 5 timed HetA's FlexAttention implementation against [DENSE BASELINE], reporting 1.63x block speedup for 5B; the updated 1.30x is against dense FlashAttention with fresh routing at every block. The 2.02x kernel improvement comes from comparing the two HetA kernels under identical conditions.

Complete DiT latency and overhead breakdown. On Wan2.2-5B:

MethodComplete DiT denoising, median (s)
Dense DMD26.34
Dense DMD39.52
HetA38.15

We used 32 prompts and two seeds at 121x704x1280, alternating the methods on the same A100 (BF16, batch size one). DiT times include all steps, uncertainty prediction, routing, swaps and gathering/scattering, excluding loading and warmup.

HetA is 1.17x faster over the complete DiT sequence (95% paired interval 1.16-1.17). A separate component profile gives 3.18 s for the first dense step, self-attention of 5.60 s (DMD3) vs 4.20 s (HetA) including gather/scatter, and 0.097 s for swaps and route rebuilding across 60 sparse blocks; all are included in the DiT totals.

At larger scale, a complete 40-block Wan2.2-14B expert forward at 121x704x1280 takes 66.63 s dense versus 31.61 s with HetA (2.11x, synthetic inputs, no trained uncertainty head). Trained Wan2.1-14B timing is in our response to KGpf.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results and revisions

We thank all four reviewers for their thoughtful and constructive suggestions, which have helped us make the paper significantly stronger. We have incorporated the new results into the revised paper, with new and revised content highlighted in blue.

  • Uncertainty routing improves on matched uniform routing: full VBench is 83.28 versus 82.68 (+0.60 points), with a 95% paired prompt interval of +0.20 to +0.97, using the same neighborhood, 7% swaps and adaptation budget.
  • Higher quality and faster execution than AHA: HetA scores 83.28 versus 82.56 (+0.72 points), with a paired prompt interval of +0.36 to +1.08 and higher quality and semantic scores. After separately optimizing AHA's routing and packing with the same local Triton kernel, preliminary three-step 5B timing at equal global-query counts gives 8.13 versus 9.98 s (1.22x speedup). Details are in our response to KGpf.
  • Faster than dense DMD3 with higher VBench: with approximately 80% of queries local, HetA scores 83.28 versus 82.94 and runs 1.17x faster over the complete DiT sequence (8.15 versus 9.52 s), including the first dense step and routing. In the blinded human study, HetA is preferred or tied in 76% of comparisons against DMD2 and 64% against DMD3. Our attention kernel is 2.02x faster than FlexAttention in the same benchmark; at 14B/704p it gives 3.78x attention and 2.23x block speedup over dense FlashAttention.
  • The routing signal predicts sensitivity to global context: on disjoint prompts, uncertainty has Spearman correlation 0.359 with the whole-denoiser dense/local output change, and selected tokens capture 1.29x random expectation (95% interval 1.21-1.38). The initial cohort gives 1.26x, supporting replication.
  • At 14B, longer adaptation narrows the quality gap, with a runtime gain: trained Wan2.1-14B achieves 1.18x complete-DiT speedup (26.34 versus 30.93 s, at the 1,625-update checkpoint). Doubling adaptation from 1,625 to 3,250 generator updates raises full VBench from 82.47 to 83.25 (dense DMD3: 84.63), improving both quality and semantic scores. Separately, at 14B/704p our implementation reaches 3.78x attention and 2.23x block speedup over dense FlashAttention. Retraining at 14B is compute-intensive, so within the rebuttal period we could run only a short schedule. The dense LoRA-based student was trained for approximately 30k DMD iterations; HetA adapts its existing attention LoRA parameters with 3,250 generator updates so far, following 500 uncertainty updates, about 7% of the 5B schedule. We are extending both stages and will report the extended results in the final version.

For completeness, the paired intervals capture prompt variation for single trained models. On 5B, the measured pipeline is dominated by the 31.4 s VAE decode; with LightTAE (about 0.21 s), the same measured non-decoder times project 1.16x end-to-end speedup, close to the measured 1.17x DiT gain. HetA scores 58.92 versus 60.80 on VBench2.0. At 14B, the gap to dense narrows from 2.16 to 1.38 points as adaptation doubles, while HetA already provides 1.18x complete-DiT speedup. Training details and prompt-matched qualitative results are in Appendix B.2 and Figures 12-13.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.