acceptodds
Withdrawn from ICLR 2027

HyMoR-Attn: Granularity-Matched Repair of Omitted Attention Support for Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Block‑sparse attention has become the backbone of efficient video diffusion. A growing body of work seeks better fidelity and efficiency trade‑offs by improving the accuracy of coarse block selection. Intuitively, a more accurate selector should yield a better frontier. However, we find that coarse‑only selection can itself limit the attainable frontier. Video‑diffusion attention is inherently two‑scale. Coarse blocks efficiently capture a locally dense body while a scattered tail of high‑value interactions remains inside omitted blocks. We call this failure the *Omitted‑Micro Problem*. We further find that omission risk is hierarchically predictable. Risk is stable at the layer and head level and stable enough for offline per‑model calibration while the coordinates of valuable Microtiles remain content‑dependent and require online selection. This observation motivates a rethink of selection‑centric sparse attention. Improving coarse rankings alone cannot recover fine interactions that the execution granularity makes invisible. We introduce **H**ybrid **M**icro **O**mission **R**epair (HyMoR‑Attn), a training‑free support‑repair method built on a coarse Macro backbone with a fine‑grained Micro residual. It combines offline risk calibration to locate vulnerable layers and heads, online peak‑preserving selection to recover the omitted tail, and granularity‑matched Macro and Micro execution with exact LSE merging. Across four settings spanning two models, two tasks and 480p and 720p resolutions, HyMoR‑Attn is Pareto‑optimal among the compared methods. It achieves up to 2.06 and 2.12 speedup over Full Attention while reaching up to 33.01 and 28.46 PSNR on HunyuanVideo and Wan-2.1 respectively. The quality ceiling of coarse block sparsity is set not by the chosen budget but by the high‑value support it discards. Matching repair granularity to local interaction density raises fidelity and efficiency together.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BADmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

HyMoR-Attn proposes training-free sparse attention for video diffusion, combining a coarse Macro core with fine-grained repair of omitted interactions. It uses offline head-risk calibration, an online sampled-LSE scorer, and separate Macro/Micro kernels with an exact normalization merge. Experiments on HunyuanVideo and Wan report improved fidelity over DFSAttn at comparable or lower latency. However, the evidence does not convincingly establish the novelty or necessity of the proposed combination.

Critical strengths
  1. The paper provides a concrete, testable diagnosis of omitted interactions. The add-back experiments measure recovery of attention-output error rather than relying solely on attention visualizations. Tables 5–7 distinguish average omissions from severe omissions and suggest that a small number of Microtiles can account for much of the recoverable error. This provides a useful empirical motivation for selective repair.

  2. The reported improvements extend across multiple settings and are supported by targeted ablations. Evaluation covers two models, two tasks, and two resolutions. Ablations examine random tile selection, random head selection, and mean-pooled scoring. Together, these results provide preliminary evidence that selective fine-grained repair can improve the practical quality–latency trade-off.

Critical weaknesses
  1. Key numerical evidence is internally inconsistent. Figure 4(c) labels the online sampled-LSE scorer’s Recall@20 as 95.4%, whereas Section 4.3, the figure caption, and the corresponding row of Table 8 report 89.0%. In Table 8, 95.4% belongs to the dense-mass oracle. The plotted improvement of 82.7 percentage points also uses 95.4%, making this more consequential than an isolated typographical error. Additionally, Section 6.2.1 claims a VBench improvement of 0.07–0.11 points over DFSAttn, while Table 2 gives 0.17, 0.26, 0.23, and 0.20 points for the quality-oriented configurations. These discrepancies reduce confidence in the quantitative presentation and require reconciliation with the underlying records.
  2. The ablations do not adequately isolate support repair from additional computation or establish the contribution of granularity matching. In Table 3, the full method has density 0.1984, while removing the Micro residual reduces density to 0.1961. Consequently, this comparison does not strictly establish that the gain is independent of increased support. A control allocating the same additional budget to the coarse core is missing. Furthermore, the paper does not compare all-fine and mixed-granularity execution on an identical support set or ablate Macro promotion. The results therefore leave unresolved how much benefit comes from support selection versus the proposed execution strategy.
  3. The central “granularity trap” argument relies on an insufficiently justified fine-grained baseline. Table 1 reports 401 seconds for Tiled-16 at 20% density, compared with 315 seconds for dense attention. This establishes the inefficiency of the tested implementation, but does not demonstrate an inherent limitation of fine-grained execution. FG-Attn is cited but not evaluated, despite explicitly addressing hardware efficiency for fine-grained sparse attention.
Questions for authors
  1. Result clarification: Is the sampled-LSE Recall@20 in Figure 4(c) 89.0% or 95.4%? Please reconcile the figure with Table 8 and provide the corresponding plotting data and script. Are the heatmaps in Figure 4(b) empirical measurements or schematic illustrations? If empirical, what model, layer–head, timestep, and samples produced them?

  2. Implementation clarification: What are the per-row budget bounds and Macro promotion threshold? Does promotion introduce additional interactions, and how are these counted in realized density? How is support overlap avoided after promotion so that the LSE merge remains valid?

  3. Evaluation clarification: What are the frame counts, total denoising steps, prompt counts, and seed counts for each setting? What statistical test and pairing unit support the significance claims? Does reported density include the 12-step dense warmup, and does end-to-end timing include scoring, selection, promotion, and merging?

  4. Validation of the central claim: Does HyMoR-Attn retain its advantage over optimized FG-Attn and a coarse-core expansion control at matched density or latency? On an identical support set, how much time does mixed-granularity execution save over all-fine execution? These comparisons would materially affect my assessment of the claimed granularity-matching contribution.

Reviewer_qtkXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the "Omitted-Micro Problem": coarse block-sparse attention in video diffusion throws away a few high-value fine-grained interactions along with the rest of a low-score block. It proposes HyMoR-Attn, a training-free method that adds a small, selectively-applied fine-grained repair on top of the coarse attention, showing better quality-speed trade-offs than prior training-free baselines on HunyuanVideo and Wan2.1.

Critical strengths
  1. The paper identifies a real and well-supported problem: Top-k coarse block selection throws away small but high-value interactions inside rejected blocks, and the authors quantify this with add-back experiments (e.g., a single microtile recovering 93.5% of a discarded block's quality loss).

  2. The ablation study cleanly isolates where the gain comes from, showing at matched density that the improvement comes from recovering the right omitted interactions rather than simply computing more.

Critical weaknesses
  1. Several design choices lack ablation: the Macro/Micro tile sizes are fixed and never varied, the claimed stability of per-head omission risk is based on very few videos, and all results come from a single GPU with hardware-specific kernels, leaving generalizability unclear.

  2. The offline calibration cost (running dense attention as ground truth) is not quantified or counted toward the reported efficiency gains, and it's unclear whether calibration needs to be redone for each resolution, which would weaken the "calibrate once" claim.

Questions for authors
  1. Have you tried ablating the Macro/Micro tile sizes themselves (not just the budget)? It would help to know if the gains are sensitive to this specific choice of Q128×K96 / Q16×K16.

  2. Why are the baselines limited to training-free methods? Have you compared against any trainable sparse attention methods (e.g., VSA, NSA) to see where HyMoR-Attn stands against them?

  3. The ablation in Table 3 is only run on one setting (HyVideo t2v 480p). Does the same ranking of component contributions hold on Wan2.1 or at 720p?

  4. How much does the offline calibration itself cost (number of videos, steps, wall-clock time), and is it counted anywhere in the reported efficiency numbers? Also, do you recalibrate separately for 480p vs 720p, or reuse the same risk set across resolutions?

  5. Figure 1 is a bit hard to read — some of the small attention blocks look quite irregular and it's not clear what they're meant to represent. Could you clarify what's being shown there, or consider cleaning it up for readability?

  6. Related work cites MonarchRT, Veda, and InputStable as making similar observations about block-pooling blind spots or offline/online sparsity splits. Could you clarify more precisely what distinguishes HyMoR-Attn's contribution from these?

Reviewer_WLH4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that block-sparse attention in video diffusion transformers discards entire coarse blocks that still contain a few high-value fine-grained interactions (the "Omitted-Micro Problem"). Add-back experiments indicate that recoverable output error inside omitted blocks concentrates in a few Q16×K16 Microtiles. They also indicate that this omission risk is fairly stable at the (layer, head) level, whereas the locations of valuable Microtiles depend on content.

Critical strengths

The proposed training-free method, HyMoR-Attn, keeps a Q128×K96 block-sparse Macro core and adds a Micro residual selected from the Core's complement. On HunyuanVideoand Wan2.1, the paper reports higher fidelity to dense attention and lower latency than DFSAttn, RAPID, AdaSpa, and SVG-2.

Critical weaknesses
  1. Repair is not isolated from extra computation and kernel changes. The no-Micro ablation runs at lower density than the full method (0.1961 vs. 0.1984; Table 3) and reports no latency. Meanwhile, the Macro block shape changes from DFSAttn's 128×128 to 128×96 (Table 1). As a result, neither the 0.57 dB gain nor the latency advantage over DFSAttn (e.g., 211 → 198 s) can be cleanly attributed to repair. The scorer also appears to be excluded from "realized density." If Eq. (2) is evaluated for every K16 tile in a risk row, as the normalization in Eqs. (3)–(4) suggests, this cost would be roughly 1/16 of dense QKᵀ per risk head. That would far exceed the residual's own 0.0023 density. Separately, the all-fine Tiled-16 comparison is unspecified, while optimized fine-grained kernels already exist (FG-Attn, Durvasula et al., 2025, §§5–6). Finally, the Pareto claim relies on a single operating point per baseline. A matched-compute coarse control and a clarified fine-kernel comparison would resolve most of this.

  2. Calibration is under-specified, and the evidence for transfer is narrow. The clamp values, τ_promote, how promotion changes the support (and whether Eq. 6's disjointness is preserved), the rule for selecting high-error rows, and the aggregation over videos and steps are all undefined (§5.1.2, §5.2.1, Appendix A.11). Cross-video stability rests on two videos (Figure 4a) and a three-video snapshot (Table 8). Seed stability is claimed (§1, Appendix A.11) but never shown. Because hierarchical predictability is a stated contribution, these gaps limit both the reproducibility and the support for that claim.

  3. Evaluation details are missing, and some results are hard to reconcile. The paper does not report total sampling steps, frame counts, seeds, the type of paired test, or uncertainty estimates (§6.1, §6.2.1). This matters because the latency gains over DFSAttn are only 3–6%. The budget sweep (Appendix A.9) reports no latency, so it does not support the claimed quality–speed knob. Table 3 is labeled as a VBench 10% subset, yet its full-method row exactly matches the full-suite row in Table 2 (28.89 / 0.892 / 0.087). The p_total = 0.90 row in Table 4 matches neither table. It is therefore unclear which configuration produced each number.

Questions for authors
  1. On HunyuanVideo 480p, how do quality and latency compare if the Micro budget, scorer included, is instead spent on extra 128×96 coarse blocks on the same risk heads? What is the latency of the no-Micro row in Table 3? Is the scorer's cost counted in the reported density?
  2. Please also explain why Table 3 (subset) reproduces Table 2 (full suite) and which configuration Table 4 uses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.