- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SQuad converts pretrained video diffusion transformers to a sub-quadratic attention architecture by composing local within-window attention with global attention across corresponding positions in different windows. The two passes reuse the original queries and keys, with the local output serving as the values for the global pass. The converted model is trained through flow-matching fine-tuning followed by DMD2 distillation, which also reduces sampling to six network evaluations. The paper evaluates generation quality on Wan 2.2 5B and Wan 2.1 1.3B, alongside attention-level and transformer-level efficiency measurements.
Critical strengths
- The method offers a simple, practical way to reduce attention cost without adding parameters or requiring a custom sparse-attention kernel. The efficiency evaluation goes beyond FLOP counts: Table 5 reports a reduction in compiled single-forward DiT latency from 667 ms to 314 ms on Wan 2.2 5B. This improvement is separate from the reduction in sampling steps and provides evidence that the attention modification delivers useful wall-clock savings.
- The strongest configuration replaces attention in all 30 blocks rather than relying on remaining full-attention layers to recover quality. On Wan 2.2 5B, it achieves a VBench total of 83.20, compared with 83.08 for the original model and 82.54 for the full-attention DMD baseline. The human preference study provides additional support for the resulting generation quality. The local/global and training-stage ablations also help establish which components are necessary for this fully converted configuration.
Critical weaknesses
- The architectural contribution needs clearer positioning relative to earlier local/global attention designs. In particular, MaxViT: Multi-Axis Vision Transformer (ECCV 2022) already composes local block attention with dilated global grid attention using standard attention operators and tensor rearrangements. SQuad is not identical: its reuse of the original queries and keys, composition through the value stream, and conversion of pretrained video models are potentially meaningful differences. However, the paper does not discuss this relationship, leaving unclear which aspects constitute a new attention mechanism rather than an adaptation and distillation strategy.
- The explanation for preserving expressive power is less convincing than the complexity analysis. Appendix D establishes full receptive-field connectivity, but the effective weights remain a constrained product of two attention distributions; reachability alone does not establish that the teacher’s input-dependent interactions are preserved. The empirical support is also strongest at the selected Wan 2.2 configuration. On Wan 2.1 1.3B, the semantic score falls from 79.30 to 76.44, although the DMD-only baseline falls further, so this cannot be attributed solely to SQuad. Moreover, the sequence-length sweep in Appendix H measures FLOPs with newly selected windows, not generation quality at those lengths. These results support a useful operating point more clearly than a broadly reliable quality-preserving replacement.
Questions for authors
- How does SQuad differ from MaxViT-style block/grid attention at the operator level? In particular, what is gained by reusing the original queries and keys and composing the passes only through the value stream, rather than applying two conventional attention sublayers?
- Beyond the receptive-field argument, what evidence explains why the fixed routing preserves the interactions needed by the teacher? Is there generation-quality evidence for changing duration or resolution using the same trained checkpoint and a specified window-selection rule, rather than only the FLOPs measurements in Appendix H?
2 comments
Does n√n SQuad Attention preserve the expressive power of full attention? (Weakness 2)
No. You are correct that n√n attention cannot, and does not, preserve the expressive power of an n² teacher. If this was not sufficiently clear, we will make it explicit in the revision.
The question we are trying to investigate is whether full n² interaction is actually required for video generation. The results presented in the paper suggest that a large portion of this interaction may not be necessary. Our intention is to present these results as evidence in support of this hypothesis, rather than as conclusive proof.
That said, SQuad preserves a true softmax operation within every block and contains no linearized attention approximation. We believe this provides stronger expressive power than linear attention methods, which often rely on hybrid architectures to recover expressivity (for example, see [10]).
Does quality hold beyond the teacher's token-length setting? (Question 2)
Thank you for this suggestion. To evaluate the transferability of pretrained SQuad models to different spatial and temporal resolutions at scale, we experimented with an early version of the Wan2.1-14B DMD model trained using a SQuad window size of 21×2×4.
Although the default resolution of the model is 81×480×832, we also tested inference at 121×480×832 and 81×704×1280. For the increased spatial resolution, we found that keeping the pretrained window size produced the best visual quality. Adjusting the window according to the √n rule and changing it to 21×3×4 resulted in visible content repetition.
In contrast, for increased generation length, adjusting the window size to 31×2×4 produced better results. Using the original 21×2×4 window led to visible discontinuities between the 21st and 22nd latent frames.
These results demonstrate both that transferrability beyond pretrained resolution is possible and that the spatial and temporal axes behave differently in SQuad. This observation is consistent with the conclusions drawn from the window-characterization experiments reported in the submitted manuscript. It also suggests that adaptation rules for spatial and temporal dimensions should be considered separately when transferring pretrained SQuad models to new generation settings.
[1] Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y. (2022). MaxViT: Multi-axis vision transformer. In European Conference on Computer Vision (ECCV).
[2] Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv:1904.10509.
[3] Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019). Axial attention in multidimensional transformers. arXiv:1912.12180.
[4] Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv:2004.05150.
[5] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (ICCV).
[6] Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., & Gao, J. (2021). Focal self-attention for local-global interactions in vision transformers. In Advances in Neural Information Processing Systems (NeurIPS).
[7] Bertasius, G., Wang, H., & Torresani, L. (2021). Is space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML).
[8] Zhang, M., Arora, S., Chalamala, R., Wu, A., Spector, B., Singhal, A., Ramesh, K., & Ré, C. (2024). LoLCATs: On low-rank linearizing of large language models. arXiv:2410.10254.
[9] Ghafoorian, M., Korzhenkov, D., & Habibian, A. (2025). Attention Surgery: An efficient recipe to linearize your video diffusion transformer. arXiv:2509.24899.
[10] Ghafoorian, M., & Habibian, A. (2026). ReHyAt: Recurrent hybrid attention for video diffusion transformers. arXiv:2601.04342.
Thank you for your time, effort, and feedback. We are glad that you find SQuad simple and practical, and that you find our efficiency analysis beyond FLOP counts insightful. We address your concerns below.
How is SQuad positioned relative to MaxViT-style multi-axis attention? (Weakness 1, Question 1)
Thank you for raising MaxViT [1]. As you already pointed out, our formulation differs from its block and grid attention. These differences make SQuad a distinct and novel operator. We should have discussed this connection more clearly in the paper, and we will make this and other sources of inspiration explicit in the revision.
Other local-global attention works. MaxViT is not the only work that inspired us. Sparse Transformers [2] also factorize attention into a local pass and a strided pass over every ~√n-th position, resulting in O(n√n) complexity for images. Axial Attention [3] attends along each axis in turn, while Longformer [4] combines sliding-window attention with a small number of global tokens. Swin [5] connects neighboring windows through shifting, and Focal Attention [6] combines fine-grained local attention with coarse global summaries. For video, TimeSformer [7] factorizes attention across space and time. In the context of converting pretrained attention into efficient forms through distillation, LoLCATs [8] and Attention Surgery [9] are also closely related precedents, if we ignore their linear-attention branch.
What sets our work apart? We perform a principled analysis of local and global extents. Balancing the cost of the two passes yields a window size of √n, which in turn produces the O(n√n) complexity. The derivation is provided in the supplementary material. In contrast, MaxViT fixes both extents at 7×7 [1], resulting in linear complexity but restricting attention to a fixed neighborhood regardless of input size. We intentionally accept a higher asymptotic cost so that each local pass remains a dense softmax over a window that grows with the video size.
In addition, we study large video generation models rather than image classifiers or video understanding models. In this post-training setting, even subtle changes in window shape can have a noticeable effect on performance. We demonstrate this both in the supplementary videos and through quantitative evaluations in the submitted manuscript. We therefore believe that these findings are relevant to the community.
What is gained by reusing the queries and keys and composing through the value stream? (Question 1)
Thank you for this question. Reusing queries and keys is particularly beneficial in the post-training distillation setting. Introducing two separate sublayers would require new parameters that are not present in the teacher model, creating additional training requirements.
In the context of pretraining from scratch, we agree that either formulation (shared projections versus separate sublayers) could work equally well. Our primary focus is the proposed sub-quadratic O(n√n) framework rather than a particular implementation of it.