acceptodds
Withdrawn from ICLR 2027

SQuad: Sub-Quadratic Attention Distillation for Efficient Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video Diffusion Transformers (DiTs) spend most of their compute inside the Self-Attention operation, whose cost grows quadratically, , with the number of latent tokens . For the task of video generation, the token count is large, so this term dominates runtime and memory, and thereby caps the resolution and duration we can generate. Linear and low-rank surrogates of Self-Attention trade the full softmax for cheaper kernels, but rarely recover the original's expressivity, leaving a stubborn quality gap. Motivated by this, we propose SQuad, a Sub-Quadratic Attention Distillation framework that achieves a complexity of in the resulting distilled Attention, naturally balancing the efficiency v/s expressivity trade-off. Instead of training our own Video DiT from scratch, which is prohibitively expensive, we fit a pretrained full softmax Self-Attention DiT into our proposed SQuad-Attention one by distilling the former in two stages: Flow-Matching Supervised Fine-Tuning (SFT), followed by improved Distribution Matching Distillation (DMD2) which additionally makes the sampling more efficient. On the Wan 2.2 5B text-to-video model, SQuad matches the quadratic teacher on VBench ( v/s ) while cutting the per-step per-block attention FLOPs by and attention latency by , and end-to-end DiT latency by 2, all while also generating a video in only Neural Functional Evaluations (NFEs) instead of the default .

Primary area:
generative models
Keywords:
Efficient Self-Attention, Sub-Quadratic Complexity, Efficient Video Generation, Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KbwNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SQuad converts pretrained video diffusion transformers to a sub-quadratic attention architecture by composing local within-window attention with global attention across corresponding positions in different windows. The two passes reuse the original queries and keys, with the local output serving as the values for the global pass. The converted model is trained through flow-matching fine-tuning followed by DMD2 distillation, which also reduces sampling to six network evaluations. The paper evaluates generation quality on Wan 2.2 5B and Wan 2.1 1.3B, alongside attention-level and transformer-level efficiency measurements.

Critical strengths
  1. The method offers a simple, practical way to reduce attention cost without adding parameters or requiring a custom sparse-attention kernel. The efficiency evaluation goes beyond FLOP counts: Table 5 reports a reduction in compiled single-forward DiT latency from 667 ms to 314 ms on Wan 2.2 5B. This improvement is separate from the reduction in sampling steps and provides evidence that the attention modification delivers useful wall-clock savings.
  2. The strongest configuration replaces attention in all 30 blocks rather than relying on remaining full-attention layers to recover quality. On Wan 2.2 5B, it achieves a VBench total of 83.20, compared with 83.08 for the original model and 82.54 for the full-attention DMD baseline. The human preference study provides additional support for the resulting generation quality. The local/global and training-stage ablations also help establish which components are necessary for this fully converted configuration.
Critical weaknesses
  1. The architectural contribution needs clearer positioning relative to earlier local/global attention designs. In particular, MaxViT: Multi-Axis Vision Transformer (ECCV 2022) already composes local block attention with dilated global grid attention using standard attention operators and tensor rearrangements. SQuad is not identical: its reuse of the original queries and keys, composition through the value stream, and conversion of pretrained video models are potentially meaningful differences. However, the paper does not discuss this relationship, leaving unclear which aspects constitute a new attention mechanism rather than an adaptation and distillation strategy.
  2. The explanation for preserving expressive power is less convincing than the complexity analysis. Appendix D establishes full receptive-field connectivity, but the effective weights remain a constrained product of two attention distributions; reachability alone does not establish that the teacher’s input-dependent interactions are preserved. The empirical support is also strongest at the selected Wan 2.2 configuration. On Wan 2.1 1.3B, the semantic score falls from 79.30 to 76.44, although the DMD-only baseline falls further, so this cannot be attributed solely to SQuad. Moreover, the sequence-length sweep in Appendix H measures FLOPs with newly selected windows, not generation quality at those lengths. These results support a useful operating point more clearly than a broadly reliable quality-preserving replacement.
Questions for authors
  1. How does SQuad differ from MaxViT-style block/grid attention at the operator level? In particular, what is gained by reusing the original queries and keys and composing the passes only through the value stream, rather than applying two conventional attention sublayers?
  2. Beyond the receptive-field argument, what evidence explains why the fixed routing preserves the interactions needed by the teacher? Is there generation-quality evidence for changing duration or resolution using the same trained checkpoint and a specified window-selection rule, rather than only the FLOPs measurements in Appendix H?
2 comments

Thank you for your time, effort, and feedback. We are glad that you find SQuad simple and practical, and that you find our efficiency analysis beyond FLOP counts insightful. We address your concerns below.

How is SQuad positioned relative to MaxViT-style multi-axis attention? (Weakness 1, Question 1)

Thank you for raising MaxViT [1]. As you already pointed out, our formulation differs from its block and grid attention. These differences make SQuad a distinct and novel operator. We should have discussed this connection more clearly in the paper, and we will make this and other sources of inspiration explicit in the revision.

Other local-global attention works. MaxViT is not the only work that inspired us. Sparse Transformers [2] also factorize attention into a local pass and a strided pass over every ~√n-th position, resulting in O(n√n) complexity for images. Axial Attention [3] attends along each axis in turn, while Longformer [4] combines sliding-window attention with a small number of global tokens. Swin [5] connects neighboring windows through shifting, and Focal Attention [6] combines fine-grained local attention with coarse global summaries. For video, TimeSformer [7] factorizes attention across space and time. In the context of converting pretrained attention into efficient forms through distillation, LoLCATs [8] and Attention Surgery [9] are also closely related precedents, if we ignore their linear-attention branch.

What sets our work apart? We perform a principled analysis of local and global extents. Balancing the cost of the two passes yields a window size of √n, which in turn produces the O(n√n) complexity. The derivation is provided in the supplementary material. In contrast, MaxViT fixes both extents at 7×7 [1], resulting in linear complexity but restricting attention to a fixed neighborhood regardless of input size. We intentionally accept a higher asymptotic cost so that each local pass remains a dense softmax over a window that grows with the video size.

In addition, we study large video generation models rather than image classifiers or video understanding models. In this post-training setting, even subtle changes in window shape can have a noticeable effect on performance. We demonstrate this both in the supplementary videos and through quantitative evaluations in the submitted manuscript. We therefore believe that these findings are relevant to the community.

What is gained by reusing the queries and keys and composing through the value stream? (Question 1)

Thank you for this question. Reusing queries and keys is particularly beneficial in the post-training distillation setting. Introducing two separate sublayers would require new parameters that are not present in the teacher model, creating additional training requirements.

In the context of pretraining from scratch, we agree that either formulation (shared projections versus separate sublayers) could work equally well. Our primary focus is the proposed sub-quadratic O(n√n) framework rather than a particular implementation of it.

Does n√n SQuad Attention preserve the expressive power of full attention? (Weakness 2)

No. You are correct that n√n attention cannot, and does not, preserve the expressive power of an n² teacher. If this was not sufficiently clear, we will make it explicit in the revision.

The question we are trying to investigate is whether full n² interaction is actually required for video generation. The results presented in the paper suggest that a large portion of this interaction may not be necessary. Our intention is to present these results as evidence in support of this hypothesis, rather than as conclusive proof.

That said, SQuad preserves a true softmax operation within every block and contains no linearized attention approximation. We believe this provides stronger expressive power than linear attention methods, which often rely on hybrid architectures to recover expressivity (for example, see [10]).

Does quality hold beyond the teacher's token-length setting? (Question 2)

Thank you for this suggestion. To evaluate the transferability of pretrained SQuad models to different spatial and temporal resolutions at scale, we experimented with an early version of the Wan2.1-14B DMD model trained using a SQuad window size of 21×2×4.

Although the default resolution of the model is 81×480×832, we also tested inference at 121×480×832 and 81×704×1280. For the increased spatial resolution, we found that keeping the pretrained window size produced the best visual quality. Adjusting the window according to the √n rule and changing it to 21×3×4 resulted in visible content repetition.

In contrast, for increased generation length, adjusting the window size to 31×2×4 produced better results. Using the original 21×2×4 window led to visible discontinuities between the 21st and 22nd latent frames.

These results demonstrate both that transferrability beyond pretrained resolution is possible and that the spatial and temporal axes behave differently in SQuad. This observation is consistent with the conclusions drawn from the window-characterization experiments reported in the submitted manuscript. It also suggests that adaptation rules for spatial and temporal dimensions should be considered separately when transferring pretrained SQuad models to new generation settings.

[1] Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y. (2022). MaxViT: Multi-axis vision transformer. In European Conference on Computer Vision (ECCV).

[2] Child, R., Gray, S., Radford, A., & Sutskever, I. (2019). Generating long sequences with sparse transformers. arXiv:1904.10509.

[3] Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019). Axial attention in multidimensional transformers. arXiv:1912.12180.

[4] Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv:2004.05150.

[5] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (ICCV).

[6] Yang, J., Li, C., Zhang, P., Dai, X., Xiao, B., Yuan, L., & Gao, J. (2021). Focal self-attention for local-global interactions in vision transformers. In Advances in Neural Information Processing Systems (NeurIPS).

[7] Bertasius, G., Wang, H., & Torresani, L. (2021). Is space-time attention all you need for video understanding? In International Conference on Machine Learning (ICML).

[8] Zhang, M., Arora, S., Chalamala, R., Wu, A., Spector, B., Singhal, A., Ramesh, K., & Ré, C. (2024). LoLCATs: On low-rank linearizing of large language models. arXiv:2410.10254.

[9] Ghafoorian, M., Korzhenkov, D., & Habibian, A. (2025). Attention Surgery: An efficient recipe to linearize your video diffusion transformer. arXiv:2509.24899.

[10] Ghafoorian, M., & Habibian, A. (2026). ReHyAt: Recurrent hybrid attention for video diffusion transformers. arXiv:2601.04342.

Reviewer_3iUMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SQuad, a sub-quadratic attention distillation framework for efficient video generation. It replaces full self-attention with sequential local attention within windows and global attention across corresponding positions in different windows. By reusing queries and keys and feeding the local attention output into the global attention’s value input, SQuad preserves softmax attention and a global receptive field within a single layer while reducing attention complexity from O(n2)O(n^2) to O(nn)O(n\sqrt n). The authors combine flow-matching supervised fine-tuning with DMD2 distribution-matching distillation to adapt pretrained video DiTs to the new structure and reduce the number of network evaluations required for sampling. Experiments demonstrate a competitive trade-off between generation quality and computational efficiency.

Critical strengths
  1. Simple and efficient design. SQuad reduces attention complexity from O(n2)O(n^2) to O(nn)O(n\sqrt n) while retaining a global receptive field within one layer. It introduces no additional parameters and uses standard attention operators, making it practical for adapting pretrained models.

  2. Demonstrated quality–efficiency trade-off. On Wan 2.2 5B, SQuad maintains comparable overall generation quality while achieving the lowest single-forward DiT latency among the evaluated methods, including a 1.36× compiled speedup over the fastest competing baseline.

Critical weaknesses
  1. The speed advantage has not been established at comparable quality to the strongest baselines. SQuad achieves the lowest single-forward DiT latency, but its VBench score is lower than those of Radial Attention, Jenga, and VSA. It remains unclear whether SQuad can retain a substantial speed advantage when matching the generation quality of these stronger baselines.

  2. Similar aggregate VBench scores do not imply uniform quality preservation. On Wan 2.2 5B, Dynamic Degree drops from 65.28 to 57.22, raising concerns about reduced motion. Wan 2.1 1.3B also shows declines in semantic quality, particularly spatial relationships and scenes. These results warrant a more qualified claim of near-lossless quality preservation.

  3. Generation quality on larger models remains unverified. Distillation and generation quality evaluations are limited to Wan 2.2 5B and Wan 2.1 1.3B. Experiments on Wan 2.1 14B measure computational efficiency only, leaving uncertainty about whether the method maintains a favorable quality–efficiency trade-off at larger model scales.

Questions for authors
  1. Could you compare inference speed at comparable generation quality to the strongest baselines, or generation quality at matched inference latency, to establish whether SQuad offers a better quality–efficiency trade-off?

  2. What explains the decrease in Dynamic Degree from 65.28 to 57.22 on Wan 2.2 5B? Is this accompanied by visibly reduced motion, particularly for prompts requiring substantial subject or camera movement?

  3. Could you provide generation quality and inference efficiency results after distillation on a larger model, such as Wan 2.1 14B? If this is infeasible, please clarify whether the limitation stems from compute or time constraints, training instability, or difficulties in recovering quality, as this distinction would affect the assessment of scalability.

1 comment

Thank you for your review. We are glad that you find SQuad simple and efficient. We appreciate your concerns and address them below.

Speed at matched quality (Weakness 1, Question 1)

Thank you for raising this point. We emphasize that VBench is only one part of the evidence. We use VBench as a necessary, but not sufficient, test of whether SQuad attention can scale, since the benchmark covers only a subset of quality dimensions.

For perceived quality, we primarily rely on the user study (Tab. 2) and the extensive qualitative videos provided in the supplementary material. These qualitative evaluations show that our videos are on par with both the DMD teacher and Radial Attention, which is the baseline with the highest VBench score. We believe that the combination of quantitative and qualitative results demonstrates that quality is preserved.

A quality-versus-efficiency Pareto frontier is a common way to frame results in this area. However, we use the best recommended operating point for each baseline. In addition, we do not use warmup steps or warmup transformer blocks for any of them. Further improving their latency would therefore go beyond the scope of this work.

As a result, constructing a Pareto curve would mainly require retraining our SQuad models with less optimal local-global attention splits, leading to complexity worse than O(n√n). We have already conducted a related experiment (Tab. 6) while studying the importance of window shape. Unfortunately, repeating the same ablation and identifying optimal window shapes for multiple alternative splits is not feasible within the available time. We therefore prefer to compare each method at its best operating point.

We would also like to emphasize that our main contribution is a different perspective on efficiency: a simple and easy-to-implement operator, rather than a complex hybrid architecture or a hand-tuned, training-free attention kernel. This makes SQuad significantly more deployment-friendly on non-NVIDIA GPUs that lack CUDA and Triton support, as well as on mobile NPUs.

Per-dimension quality (Weakness 2, Question 2)

We agree that an aggregate score can obscure changes in individual quality dimensions. This is why we report all individual dimensions alongside the overall score.

It is easy to see that SQuad outperforms the baselines on some dimensions while being slightly worse on others. For this reason, we believe that the user study provides a more meaningful comparison between models, with VBench serving a complementary role. As reported in the paper, participants did not express a decisive preference for any particular model, suggesting that the overall quality is comparable.

As shown in the supplementary videos, outputs produced by SQuad do not appear static, nor do they create the impression of reduced motion.

Larger models (Weakness 3, Question 3)

During the discussion period, we conducted a preliminary training study on the Wan2.1-14B model using SQuad attention in all transformer blocks at a resolution of 81×480×832. All experiments used LoRA training with a rank of 512.

Consistent with the findings reported in the submitted manuscript, we observed that starting DMD training directly from the open-source checkpoints with SQuad attention produces inferior results. In particular, the boundaries between windows remain clearly visible. We therefore first performed an SFT stage on the same dataset for 2,000 steps using a global batch size of 4. We then continued training the same LoRA adapter using the DMD objective.

Using a window size of 21×2×4, we obtained a VBench score of 83.51 after 600 DMD training updates. The generated samples look qualitatively strong and do not exhibit significant artifacts.

While these results are preliminary, they suggest that the conclusions presented in the submitted manuscript also extend to larger-scale models.

Reviewer_3CDGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

SQuad replaces the full softmax self-attention in a pretrained video DiT with a composition of two ordinary softmax attentions: a local pass within windows of about n\sqrt{n} tokens, followed by a global pass across windows at the same within-window position, with the locally mixed values feeding the global pass. This gives every token a full receptive field in a single layer at O(nn)O(n\sqrt{n}) attention cost, with no additional parameters. A pretrained Wan model is fitted to the new attention in two stages — flow-matching SFT, then DMD2 step distillation — which also reduces sampling from 100 to 6 NFEs. On Wan 2.2 5B, SQuad reports VBench 83.20 vs. 83.08 for the original model, with substantially lower attention FLOPs and lower measured per-forward latency than the compared efficient-attention methods, all evaluated at 6 NFEs.

Critical strengths
  1. The operator itself is simple, correct, and practically appealing. Both passes are standard softmax attentions separated by re-indexing, so SQuad needs no new parameters, no learned token selection, and no bespoke CUDA kernel. The complexity analysis (Appendix E–F, with w⋆=nw^\star = \sqrt{n} balancing the two passes) and the receptive-field argument (Appendix D) are correct for what they claim, and the measured FLOPs track the theoretical O(nn)O(n\sqrt{n}) curves closely (Appendix H). It is also expected and reassuring that, within the authors' own pipeline, SQuad's per-forward latency is lower than the full-attention DMD model's — an attention factorization that did not achieve at least this would have little purpose. The 20-block vs. 30-block ablation contrast (Tables 3, 7; Appendix I) is furthermore honestly reported and informative about when both passes and both training stages become necessary.
Critical weaknesses
  1. Running every baseline at a unified 6 NFEs is likely unfair to the baselines, so the comparative efficiency claims are not established. All efficient-attention baselines in Tables 1 and 4–5 are evaluated at 6 NFEs on Wan 2.2 5B, yet none of these methods was designed or released in that form: Radial Attention and Jenga are training-free methods prescribed for ~50 steps (Radial additionally requires dense attention for the first 12 timesteps and the first block, which is impossible within a 6-step budget), VSA's released Wan checkpoints run at 50 steps, and Attention Surgery / ReHyAt have no public code or Wan 2.2 5B checkpoints at all (their original quality-optimal configurations convert 15 of 30 blocks on Wan 2.1 1.3B, whereas the supplement's parameter counts suggest up to 30 converted blocks here). The paper does not state whether these are official implementations or the authors' ports, how much each was (re)trained on Wan 2.2 5B, how their prescribed protocols were adapted to 6 steps, or whether their custom kernels (VSA's sparse kernel, Radial's FlashInfer backend) were used for latency. This matters because the latency results suggest they were not: Radial Attention is O(nlog⁡n)O(n \log n) yet measures almost as slow as dense attention in Table 5 (619 ms vs. 667 ms compiled), which is difficult to reconcile with its official implementation. Measuring kernel-dependent methods in a kernel-free harness, on a step-distilled backbone they were not designed for, and with undisclosed re-training, substantially weakens the headline claim of being faster than prior efficient-attention methods. I want to be clear that I do not see evidence of intentional misrepresentation — the reported baseline VBench scores are in fact higher than SQuad's — but as reported, the comparison cannot support the paper's central comparative conclusions, and it also weakens the apparent novelty/effect size of the method.

  2. There is no evaluation of the attention acceleration without step distillation, leaving it unclear whether SQuad works outside the DMD2 regime. Every headline comparison is performed at 6 NFEs after DMD2, so the effect of the attention factorization is entangled with distribution-matching step distillation. The only internal evidence without DMD2 points the other way: with all 30 blocks converted, SFT alone at 100 NFEs reaches only 73.03 VBench, while DMD alone reaches 80.91 and the combination 82.99 (Table 3), i.e., much of the quality recovery in the current recipe is done by DMD2 rather than by fitting the new attention itself. The authors acknowledge this coupling in the limitations, but for a paper whose contribution is an attention operator, the missing experiment is central rather than peripheral: SQuad fitted with flow-matching only and run at the original step count, compared at the same sampler and training budget against the full-attention model and against other efficient-attention baselines. Without it, I cannot tell whether SQuad is an effective attention acceleration in its own right — usable, for example, in a standard many-step Wan model — or whether its quality preservation depends on being inside a few-step distilled model. This substantially limits the generality and significance of the contribution as demonstrated. Even within the 6-NFE regime, the "matches the teacher" claim holds only in aggregate: on Wan 2.2 5B, SQuad's Dynamic Degree drops to 57.22 from the original's 65.28 (while the full-attention DMD model reaches 75.83), suggesting its videos are noticeably more static, and on Wan 2.1 1.3B the Semantic score drops from 79.30 to 76.44 and Scene from 52.71 to 45.62 (Table 1). With no variance reported, headline differences such as 83.20 vs. 83.08 are also difficult to assess. A per-dimension reading therefore gives a less favorable picture than the total score, and strengthens the concern that quality preservation is specific to this distilled, aggregate-metric setting.

Questions for authors
  1. For each efficient-attention baseline (VSA, Jenga, Radial Attention, Attention Surgery, ReHyAt): was the official code/checkpoint used, or the authors' re-implementation/port to Wan 2.2 5B? How was each adapted to 6 NFEs (e.g., how was Radial Attention's dense-warmup prescription handled), how much was each (re)trained and on what data, how many blocks were converted where applicable, and were the methods' official custom kernels used in the Tables 4–5 latency measurements? A per-baseline provenance table would be very helpful.

  2. Without step distillation, how well does SQuad work? Specifically, what are the quality and latency of the SFT-fitted SQuad model at the original step count (e.g., 50/100 NFEs), and how do they compare with the full-attention model and with at least one strong efficient-attention baseline run at the same step count and comparable training? If SFT alone (73.03 in Table 3) is simply under-trained, a longer-SFT result would address this.

  3. Relatedly, is the intended scope of SQuad "an attention operator for few-step distilled generators" rather than a general replacement for full attention in video DiTs? If the latter, what evidence supports quality preservation as the sampler moves away from the 6-NFE DMD2 setting?

1 comment

We thank the reviewer for their effort and address the expressed concerns below.

Weakness 1, Question 1

We agree that most efficient baselines were originally released as multi-step models. However, motivated by the practical need for lower-cost inference, we evaluated them in the few-step regime. We found that, when applied to the pretrained DMD checkpoint, Jenga and Radial Attention perform well even without training, as noted by the reviewer. Importantly, the reported quality was achieved without any warmup blocks or warmup denoising steps.

Regarding the claimed O(n log n) complexity of Radial Attention, this is an asymptotic estimate. At our target resolution, it can still be more expensive than SQuad, whose complexity is O(n√n). Our analytical FLOPs analysis (Tab. 4) confirms that SQuad is cheaper at this resolution. In addition, because Radial Attention relies on the flashinfer backend, we could not compile it with the "max-autotune" mode used for the other methods in the table and instead used the "default" mode. This likely explains its smaller compilation gains relative to the other baselines.

For Radial Attention, we used a tile size of 64, a decay factor of 0.8, and the flashinfer backend. We did not use SageAttention, either for Radial Attention or for any other experiments. For Jenga, we used the vsa library (v0.0.4), a self-attention drop rate of 0.8, a cumulative-probability threshold of 0.9, and a tile size of 64. We also did not enforce attention to first-frame tokens, as we found no impact on quality.

Although Jenga was originally proposed as a training-free method, we found that enabling Jenga during DMD training yields even better performance, as shown in Tab. 4. For all experiments that required training, we used the same DMD training protocol as for SQuad. The only exception was VSA, for which we reduced the learning rate to 4e-6 (and to 2e-6 for fake-score training), since higher values caused divergence. For VSA, we used the implementation from fastvideo-kernel (v0.3.2) with a tile size of 4×4×4.

To replace all 30 transformer blocks in the ReHyAt experiments, we implemented the Decoupled-DMD training pipeline following the setup described in Ghafoorian et al., MobileWan: Closing the Quality Gap for Mobile Video Diffusion (2026).

Weakness 2, Questions 2 and 3

We acknowledge in the manuscript that, in the post-training regime, SFT training alone was not sufficient to match the quality of the multi-step baseline. Therefore, whether SQuad Attention can serve as a drop-in replacement for full bidirectional self-attention remains an open question.

At the same time, we believe this question should be viewed in the context of the overall objective. Our goal is not to claim that SQuad universally replaces full attention under all training regimes. Rather, we aim to demonstrate that, in the practically important few-step setting, SQuad provides a simple and effective solution for efficient inference while maintaining competitive quality.

We believe this simplicity is a significant advantage for the community. Many efficient attention methods rely on specialized kernels, custom hardware support, or increasingly complex hybrid architectures. In contrast, SQuad is straightforward to implement and deploy. For example, it facilitates deployment on mobile NPUs, where only a limited set of operators is typically supported. Likewise, it is more suitable for non-NVIDIA accelerators that do not support CUDA or Triton kernels.

For these reasons, we view SQuad not only as an efficient attention mechanism but also as a deployment-friendly alternative that lowers the engineering burden of bringing video generation models to a broader range of hardware platforms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.