acceptodds
Under review as a conference paper at ICLR 2027

Towards a Low-Delta Oracle Policy for Improved Upper Bound in Sparse Attention

Abstract

While Diffusion Transformers (DiTs) deliver remarkable visual quality in video generation, the high cost of 3D spatio-temporal attention hinders their scalability. To evaluate and optimize sparse attention, existing studies rely on the Top- Oracle Policy. However, the rigid truncation of this approach discards the continuous tail, introducing structural errors that accumulate during iterative diffusion. To address these limitations, we introduce a novel Low-Delta Oracle Policy based on a proof that sparse attention achieves zero error when grouping identical attention scores. Building on this, our oracle analysis shows that leveraging low-delta scores preserves the structural integrity of the distribution. Based on this observation, we propose a Unified Tail Aggregation (UTA) method, which aggregates logits where the score variance is bounded by a marginal delta, seamlessly integrating with efficient attention kernels while incurring less than 1% FLOP overhead in our evaluation.Extensive evaluations show our approach achieves up to a 97.4% MSE reduction and a 39.6% improvement under the Top- oracle and an advanced sparse baseline, respectively.

Primary area:
optimization
Keywords:
sparse attention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eoh3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the substantial computational overhead of 3D spatio‑temporal attention in video generation. It points out that conventional Top‑K/Top‑P oracle policies adopt hard truncation and discard the continuous tail of the attention distribution, which introduces structural errors that accumulate across diffusion iterations. The authors propose the Low‑Delta Oracle Policy and theoretically prove that sparse attention can achieve zero error when aggregating tokens with identical attention scores. Built upon this theory, they design the Unified Tail Aggregation (UTA) method to aggregate logits with very small score differences. UTA can seamlessly integrate with the FlashInfer high‑performance attention kernel with less than 1% additional FLOP overhead. Experiments on Wan and HunyuanVideo text‑to‑video models show that UTA substantially reduces the MSE of attention outputs, effectively preserves the structural integrity of attention distributions, and noticeably improves video generation quality.

Critical strengths
  1. Solid theoretical contributions: The paper puts forward the zero‑error aggregation theorem for sparse attention. It identifies renormalization distortion and discarded‑tail probability loss as intrinsic drawbacks of the traditional Top‑K oracle. The Low‑Delta Oracle serves as a new theoretical upper‑bound, refreshing the evaluation thinking for sparse attention with prominent theoretical novelty.
  2. Practical and efficient engineering solution: The proposed UTA algorithm is easy to implement and can be directly plugged into existing sparse‑attention frameworks including FlashInfer and SVG2, with extra FLOP overhead below 1%. Improvements are validated in both oracle simulation and real‑world video‑generation settings. Sufficient quantitative and qualitative evidence is provided via attention‑distribution plots and attention‑map visualizations.
Critical weaknesses
  1. Despite its novelty, the overall field‑wide impact is limited. Plenty of existing sparse‑attention works for DiT‑based video generation realize extreme sparsity above 90% with minor quality degradation versus dense attention. Combined with kernel optimizations, those approaches can achieve speed‑ups of over 2×. The comparative advantages of the proposed method are not obvious.
  2. Experimental evaluation can be further strengthened: the set of compared baselines is limited; detailed breakdowns for VBench metrics are missing.
Questions for authors
  1. Compared with alternative approaches that achieve extremely high sparsity and large speed‑up ratios, is the proposed method mainly distinguished by theoretical insights while having limited practical value?
  2. Could the authors supply comparisons against more baselines?
  3. Can the authors report per‑dimension detailed scores for VBench?
Reviewer_gL3KOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the error caused by dropping low-score tokens in sparse attention. It introduces a Low-Delta Oracle Policy and Unified Tail Aggregation (UTA), which represents discarded tokens using mean keys and values with a correction for group size. The method requires no training and improves attention approximation and some video-quality metrics when added to existing sparse attention methods.

Critical strengths
  1. The method is simple and practical. It restores both the softmax normalization term and the discarded value contribution without additional training.

  2. The experimental results are impressive. UTA reduces attention approximation error across several sparse attention methods and improves some video-quality metrics without retraining.

Critical weaknesses
  1. The differences from closely related work are not clear. SVG-EAR also uses mean keys and values to compensate for skipped attention blocks. Given the same clusters and mask, UTA’s Algorithm 1 appears mathematically equivalent to its compensation component. ClusterAttention also considers similarity in attention logits when grouping keys, while ToMe uses a log-size correction to account for merged tokens. These works are not discussed. The paper should clarify its differences in both the Low-Delta analysis and the practical algorithm, and include a direct comparison with SVG-EAR.

  2. It is unclear how the Low-Delta idea is applied in practice. The paper proposes grouping tokens with similar attention scores, but the implementation uses groups and masks provided by an existing sparse attention method. It does not clearly show how Low-Delta changes the existing sparse attention method's design.

  3. The quality gains are not consistent across models. UTA improves the reported five-dimension VBench average from 83.8 to 84.5 on Wan, but leaves it unchanged at 82.2 on HunyuanVideo despite adding about 3.1% runtime overhead. Although similarity to dense outputs improves on both models, the evidence for a general improvement in video quality is still limited. Broader cross-model evaluation would strengthen this claim.

  4. The latency analysis is incomplete. The paper reports relative generation speedups, but does not provide absolute end-to-end generation times or clearly specify what is included in the timing. It also lacks a runtime breakdown for aggregation and compensation, making the practical overhead difficult to assess.

Questions for authors
  1. How does UTA differ from SVG-EAR’s compensation, and how does the Low-Delta idea go beyond related ideas in ClusterAttention and ToMe? What practical grouping or selection decisions does it change?

  2. Can the authors explain why the reported VBench score improves on Wan but not on HunyuanVideo, and provide absolute end-to-end latency with a breakdown of UTA’s added overhead?

Reviewer_JWDzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the approximation error introduced by truncation-based sparse attention in video diffusion transformers. It proposes Unified Tail Aggregation (UTA), which supplements the retained attention entries with aggregated representations of the discarded tokens. Each representative uses an averaged value and a mean logit corrected by the logarithm of its token count, allowing the omitted contribution to enter both the attention numerator and the softmax denominator. The paper proves exactness when the aggregated logits are identical and provides an approximation bound for non-identical logits in Appendix B. Experiments report attention-output MSE reductions under Top-K and several sparse attention baselines, together with improved fidelity to full-attention generation on Wan 2.1 and HunyuanVideo. The practical implementation retains substantial acceleration, although it increases inference latency relative to SVG2-Turbo.

Critical strengths
  1. UTA addresses both the omitted value contribution and the change in softmax normalization caused by truncation. The exact aggregation identity is correct, and Appendix B usefully separates the effects of tail-mass approximation and logit–value coupling. The normalization-only ablation in Appendix C.5 supports the importance of recovering both components. This makes the method potentially useful as an inference-time addition to existing sparse attention implementations.

  2. The attention-output evaluation extends beyond SVG2 to several sparse baselines in Appendix C.7, with improvements across all reported method–density pairs. The two-model generation results also show improved similarity to full-attention outputs while preserving 1.66× and 1.91× speedups. The Wan LPIPS improvement is particularly encouraging. The depth analysis and explicit reporting of adverse head-level cases provide useful information about the method’s behavior and limitations.

Critical weaknesses
  1. PISA [1] already evaluates a zeroth-order approximation whose numerator and denominator have the same algebraic form as the multi-bin UTA correction for an identical partition. It is neither discussed nor compared against. The log-count correction also has a precedent in proportional attention [2]. These connections leave the distinct contribution of UTA unclear. Furthermore, Section 3.3.1 does not formally define an oracle objective, approximation class, or resource constraint. Improving upon Top-K truncation by adding information from discarded tokens does not establish an improved theoretical upper bound under a matched budget. This concern directly affects the paper’s central contribution claim.

  2. Figures 3 and 10 analyze post-softmax probabilities and adjacent probability differences, whereas the approximation depends on within-group logit variation. Small probabilities or small absolute probability gaps do not imply similar logits, since (z_i-z_j=\log(p_i/p_j)). The proposed explanation therefore requires measurements within the actual groups used by UTA. At the system level, Table 2 implies approximately 6.6% and 3.1% longer inference than SVG2-Turbo on Wan and Hunyuan, respectively, despite the sub-1% FLOP overhead. There is no comparison against SVG2 using the same additional latency, for example by retaining more exact blocks. The unchanged reported VBench score on Hunyuan further limits the evidence for a consistent perceptual-quality gain. These gaps prevent a clear assessment of whether UTA improves the practical tradeoff.

[1] Haopeng Li et al. PISA: Piecewise Sparse Attention Is Wiser for Efficient Diffusion Transformers. arXiv:2602.01077, February 2026, version 1.

[2] Daniel Bolya et al. Token Merging: Your ViT But Faster. ICLR 2023.

Questions for authors

Contribution relative to prior work. What is the precise methodological distinction from PISA-0th [1]? Can you isolate the effects of grouping and implementation, and compare the methods under matched runtime budgets?

  1. Can you report within-group logit variance or range, the Jensen gap, and logit–value coupling for the actual Top-K and SVG2 groups? Do these quantities predict the observed error reductions and the adverse cases in Appendix C.9?

  2. Equation (14) pools the entire discarded set into one proxy, whereas Algorithm 1 introduces a proxy for each masked cluster. Which variant produces the reported results, how are the groups constructed, and how many proxies are used? Please clarify the claimed equivalence and include the costs of constructing and applying the proxy statistics.

  3. Can you provide quality–latency curves, or a latency-matched SVG2 baseline with increased density, on both models? Please also specify the prompt counts, seeds, denoising settings, and timing protocol used for the principal generation results.

  4. How is “Attention Recall” in Figure 8 computed, and how are proxy contributions counted? For a fixed retained set, conventional retained dense-attention mass would remain unchanged after adding a proxy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.