acceptodds
Withdrawn from ICLR 2027

Grid Compressed Attention: Accelerating Video Generation Attention with Training-Free Grid KV Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Attention computation is a major bottleneck in video Diffusion Transformers because its cost grows quadratically with the sequence length. Existing training-free sparse attention methods reduce computation via sparse token selection, but the more evenly distributed attention mass in video generation limits how aggressively tokens can be discarded without losing information. We introduce Grid Compressed Attention (GCA), a training-free method that compresses attention via KV compression and query grouping. Specifically, we formulate attention compression as an entropy-regularized optimization problem and derive an efficient construction based on compact KV compressions and disjoint query groups. A local Fisher characterization of the group-logit approximation error guides the selection of representative queries. Experiments on Wan2.1-14B show favorable fidelity–latency trade-offs for GCA among the evaluated configurations. These results support query-guided KV compression as a promising training-free approach to accelerating video generation attention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5vfdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Grid Compressed Attention (GCA), a training-free acceleration method for video Diffusion Transformers based on query-aware key–value compression. It formulates linear KV compression as constrained entropy-regularized attention optimization, equivalent to minimizing reverse KL from the effective compressed attention distribution to dense attention. For fixed disjoint merge groups and weights, this yields a closed-form attention rule with the logit correction H(wc)H(w_c). Representative queries guide non-uniform KV merging separately for each query-group/key-block cell. Experiments on Wan2.1-14B T2V and I2V, with 50-step and four-step inference, report favorable quality–latency trade-offs at higher attention speedups against training-free sparse and structured baselines.

Critical strengths

1. Theoretically grounded KV compression.

The paper derives an entropy-corrected attention mechanism from a constrained optimization objective (Eqs. 8–12), providing a principled way to preserve attention mass under non-uniform KV compression. The substantial performance degradation without entropy correction (Table 2) supports its effectiveness, although its advantage over simpler group-size correction remains unverified.

2. Competitive quality–efficiency trade-offs.

GCA achieves the highest PSNR and SSIM among the compared methods at approximately 3× attention speedup on both T2V and I2V (Table 1), with up to 6.13× attention and 3.03× end-to-end speedups (Figure 1). The quality–latency curves and ablation studies further demonstrate the effectiveness of query-guided KV compression.

Critical weaknesses

1. Gap between theoretical optimization and practical implementation.

Although GCA is motivated by reverse-KL minimization (Eq. 8), its implementation relies on several heuristic approximations, including disjoint key assignment, representative queries replacing optimal attention-weighted queries, and greedy Fisher-based sampling. These approximations are not guaranteed to minimize the original objective. Moreover, the connection between local grid-cell optimization and global attention fidelity remains insufficiently validated.

2. Limited practical benefit of Fisher-guided representative selection.

Despite the theoretical analysis in Theorem 1, Fisher-guided sampling improves PSNR by only 0.21 dB over random representative selection (Table 2). The paper lacks statistical analysis across different compression ratios and denoising steps, as well as runtime ablations demonstrating whether this relatively complex selection mechanism justifies its overhead.

3. Substantial fidelity degradation relative to full attention.

Although GCA outperforms competing methods at high speedups, its absolute fidelity remains limited. At approximately 3× attention speedup, PSNR reaches only 14.59 dB for T2V and 17.08 dB for I2V, with SSIM values of 0.52 and 0.56 (Table 1). Furthermore, the reported VBench metrics cover only subject consistency and imaging quality, leaving motion consistency, temporal stability, and text-video alignment insufficiently evaluated.

4. Limited experimental generalizability and statistical validation.

Experiments are limited to Wan2.1-14B and its step-distilled variants, without validation on other video DiT architectures such as HunyuanVideo. Moreover, T2V and I2V evaluations include only 20 inputs each, and the reported VBench metrics cover only limited aspects of generation quality. This narrow evaluation makes it difficult to reliably assess overall video quality and establish the robustness of the reported improvements.

Questions for authors
  1. How do GCA's heuristic approximations relate to the reverse-KL objective (Eq. 8), and how does local optimization affect global attention fidelity?

  2. Does the modest 0.21 dB PSNR gain from Fisher-guided sampling (Table 2) justify its computational overhead?

  3. Could the authors report additional VBench metrics, including motion consistency, temporal stability, and text-video alignment?

  4. Could the authors evaluate GCA on other video DiT architectures and larger prompt sets to demonstrate generalizability?

Reviewer_Hx3YOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Grid Compressed Attention (GCA), which aims to accelerate inference in video diffusion Transformers through training-free KV compression. The method partitions queries into groups and KV into blocks, uses representative queries to guide weighted merging of KV within each block, and corrects the compressed attention scores with an entropy bias. The authors further use local Fisher geometry to explain the approximation error and select representative queries. The paper conducts text-to-video and image-to-video experiments only on Wan2.1-series models.

Critical strengths
  • Addresses an inference efficiency problem of practical significance. Long-sequence attention in video generation incurs high computational costs, and GCA can be directly applied to existing pretrained models without additional training or fine-tuning.
  • Provides a mathematical explanation for compressed attention. The paper derives a closed-form solution for the attention weights under disjoint merging through an entropy-regularized objective and characterizes the group-score approximation error using KL divergence and a local Fisher quadratic form.
Critical weaknesses
  • Limited validation of generalizability. The experiments focus on Wan2.1-series models, with only 20 samples used for each of the T2V and I2V tasks, resulting in a small evaluation scale. In addition, the paper lacks validation across model architectures, resolutions, and video lengths.
  • Insufficient evidence of generation quality preservation. The paper presents only two sets of visual examples on the first page, and the results generated by GCA differ noticeably from those generated by full attention.
  • Insufficient novelty. The core of the method is to merge key and value tokens during inference and correct the attention scores through entropy compensation, which is directly related to existing token merging and attention compensation methods. In particular, in the special case of uniform merging weights, GCA's entropy compensation has the same form as the token-count compensation in ToMe, making it difficult to assess the substantive gains of the new designs over existing methods.
Questions for authors
  1. Clarify the new contributions relative to existing merging methods.
  2. Explain the output differences in the visualizations on the first page, please explain why the proposed method and dense attention produce visibly different generation results.
  3. Expand the sample size and report the stability of the results. Please evaluate on a larger and more diverse set of prompts, report means and uncertainty, and analyze performance in scenarios involving fast motion, complex interactions, and fine textures.
  4. Supplement validation of generalization across configurations. Please validate the method on more model architectures and at different resolutions and video lengths.
Reviewer_Gt4GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Grid Compressed Attention (GCA), a training-free approximation to self-attention for video diffusion transformers. It formulates linear key–value compression through an entropy-regularized objective and derives entropy-corrected attention weights for disjoint compression supports. The practical algorithm partitions queries and keys into a grid, selects representative queries using a probe-based Fisher surrogate, and constructs query-guided compressed keys and values. Experiments on Wan2.1-14B text-to-video and image-to-video models, including 50-step and four-step variants, report favorable fidelity–latency trade-offs relative to the evaluated sparse and structured attention configurations.

Critical strengths
  1. A practically relevant problem and a coherent design.

The paper addresses the substantial cost of video attention with a training-free approach that can be applied to pretrained models. Its mathematical formulation provides a useful rationale for the compression mechanism rather than relying entirely on an empirical construction.

  1. Encouraging performance and informative component evidence.

The reported comparisons support a favorable speed–fidelity trade-off in the evaluated settings. The component ablations also provide meaningful evidence for the importance of entropy correction and query-guided merging. 51947_Grid_Compressed_Attentio

Critical weaknesses
  1. The quality evaluation is too limited to establish a robust practical fidelity–latency trade-off.

    Section 5.1 evaluates only 20 text prompts and 20 image–prompt pairs, all within the Wan2.1-14B family. Table 1 reports two VBench dimensions, subject consistency and imaging quality, but does not establish preservation of motion quality or prompt adherence. PSNR, SSIM, and LPIPS against same-seed dense outputs measure reference fidelity, but do not by themselves resolve these broader quality questions. The paper also lacks uncertainty estimates across prompts and generation seeds. Consequently, the results provide useful evidence for the tested configurations, but limited support for the robustness of the reported frontier across diverse video content.

  2. The systems evaluation omits essential information about hardware and peak-memory costs.

    The manuscript reports attention and end-to-end latency without specifying the GPU model/count, batch size, software versions, or a sufficiently detailed timing protocol. Appendix G explicitly reports modeled global-memory reads rather than measured HBM traffic or peak allocation. This distinction matters because GCA constructs different compressed KV representations for different query groups: materializing all cells would produce N^2 C / (Bq Bk) representatives per head. The manuscript does not explain the buffering, streaming, or tiling strategy used to manage these intermediates. Hardware details, peak memory, and length-scaling measurements are needed to assess the reproducibility and deployment feasibility of the reported speedups.

  3. The practical representative-selection and temporal-reuse choices are not sufficiently validated.

    The conditional optima and Theorem 1 motivate the method, but do not establish that the implemented probe-based, span-greedy selection minimizes the compression objective. The paper appropriately identifies this selection as a heuristic. Table 2 compares it with random selection only for T2V at C=4, reporting a 0.210 dB PSNR gain and an LPIPS improvement from 0.58 to 0.49, without repeated random selections or a selection-specific cost comparison. There is also no ablation of reusing first-frame indices rather than selecting representatives independently per frame. The evidence therefore does not yet establish the reliability or quality–cost advantage of these choices, particularly under substantial object or camera motion.

Questions for authors
  1. Quality and coverage: How were the 20 examples for each task selected? Do the reported gains persist on a larger, predefined evaluation set and across generation seeds, including motion quality and prompt adherence? Please provide paired uncertainty estimates for the main comparisons.

  2. Systems reproducibility: What GPU configuration, batch size, software/kernel versions, warm-up, synchronization, and repetition procedures were used? What is the measured peak GPU memory for each value of C, and how are query-group-specific compressed KV intermediates buffered or streamed?

  3. Representative selection: Under comparable compute budgets, how does Fisher-based selection compare with repeated random selection and simpler deterministic alternatives? How much selection overhead is incurred, and how closely does the practical construction approximate the stated reverse-KL objective?

  4. Temporal reuse and scaling: What changes when representatives are selected separately for each frame rather than reusing first-frame indices? Do accuracy, runtime, and memory remain favorable for stronger motion, longer videos, or different resolutions and model families?

These questions concern the evidence needed to assess the current method; their answers could materially change my recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.