CoFiRA: Compute–Fidelity Region Allocation for Efficient Video Generation

Diffusion transformers have substantially advanced video generation, but their attention cost grows rapidly with the large spatiotemporal token sequences involved. Sparse attention is a promising way to alleviate this bottleneck, yet evaluated video attention remains broadly distributed even under oracle selection, suggesting diminishing returns from selector-only sparsification. Existing compensation methods recover non-exact regions with a single summary, but this binary choice between a fixed surrogate and exact attention limits the representations available for regional allocation. We introduce **CoFiRA**, a training-free framework for heterogeneous region-wise compute–fidelity allocation that couples regional representation design with compute allocation. Using query-conditioned contribution and within-cluster K/V variation, CoFiRA assigns each region a calibrated mean, a variable-capacity nested set of real K/V representatives, or exact attention, creating a scalable representation space matched to regional demand. A fidelity-aware runtime packs variable-capacity computations and schedules their irregular routes on device. On HunyuanVideo T2V 13B, CoFiRA achieves a 2.27× end-to-end speedup while reaching 33.09 dB PSNR. Code and demo samples are available at [Anonymous/CoFiRA](https://anonymous.4open.science/r/CoFiRA/).