- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the compatibility problem between 3D RoPE and a query independent low rank global branch in video Diffusion Transformers. It proposes RoLA, which applies a truncated pretrained rotary embedding after a nonlinear low rank feature map, thereby preserving genuine cross token aggregation and a reusable linear summary without additional positional parameters. Mechanism level analyses and generation experiments on Wan2.1 1.3B and 14B show that RoLA remains competitive at 90% sparsity and achieves up to 2.63 times end to end inference speedup over dense attention on an H100 GPU for 720p, 81 frame videos.
Critical strengths
- The paper identifies a clear structural tension between 3D RoPE and query independent linear attention in video DiTs. RoLA addresses this tension by applying truncated RoPE after the nonlinear low rank map, preserving relative position structure and a reusable global summary without additional positional parameters.
- The method is supported by mechanism level analyses using real Wan2.1 features and controlled positional tests. The results show that applying RoPE outside the activation preserves both position awareness and shift invariance, while the rank sweep supports the selected bottleneck dimension.
- RoLA achieves a strong quality and efficiency trade off at 90% sparsity on both Wan2.1 1.3B and 14B. It remains competitive with dense and sparse baselines while achieving a 2.63 times end to end speedup over dense attention on an H100 GPU for 14B 720p videos.
- The empirical study includes matched comparisons with VSA, SLA, SALAD, and RoPeSLR, together with controlled ablations of the global branch, rotary rank, activation function, and fine tuning protocol. The detailed algorithm, hyperparameters, complexity analysis, and reproducibility statement make the evaluation relatively transparent.
Critical weaknesses
-
The evaluation is confined to Wan2.1 1.3B and 14B under standard sequence lengths. Generalization to other video diffusion backbones, longer sequences, alternative RoPE layouts, and broader data distributions remains unestablished.
-
The primary latency measurements report DiT denoising time and exclude text encoding and VAE stages, while the deployment results in Figure 1 use an RTX 5090 and the main benchmark uses an H100 GPU. The reported end to end acceleration is therefore hardware and pipeline specific, which limits its interpretation as a full application level speedup.
-
The theoretical results characterize positional behavior and rank truncation under controlled inputs, but they do not connect operator fidelity to generation quality. The theoretical justification for the selected rank and its downstream benefits therefore remains largely empirical.
-
The quality assessment relies mainly on VBench dimensions and similarity metrics against dense attention outputs. Without human preference studies or systematic failure analysis, the effects on prompt adherence, fine grained spatial relations, and temporal consistency in difficult cases remain unclear.
Questions for authors
- The reported 2.63 times end to end speedup appears to measure DiT denoising only, and Figure 1 and the main benchmark use different GPUs. What is the speedup when measured over the complete text to video pipeline, including text encoding and VAE decoding, on a single hardware configuration?
- The rank sweep motivates the bottleneck dimension empirically, while the theory section analyzes positional behavior under controlled inputs. Is there any measurable link between operator level fidelity and downstream generation quality that would make the rank choice predictable rather than searched?
- VBench and similarity to dense outputs may miss degradations in prompt adherence, fine grained spatial relations, or temporal consistency at 90 percent sparsity. Can additional perceptual evidence be reported for these difficult cases, for example a small human preference comparison or a breakdown on motion heavy prompts?
4 comments
This response continues our preceding reply and addresses the connection between theory and empirical results (W3 & Q2), followed by quality preservation and difficult-case evaluation (W4 & Q3).
3. Connection between Theory and Empirical Results (W3 & Q2)
We acknowledge that the manuscript presents the operator-level rank sweep and downstream rank ablation separately, without clearly explaining their empirical relationship.
Proposition 2 provides structural motivation by relating the truncation error, under its idealized assumptions, to the energy in the discarded rotary subspace. It therefore motivates rank truncation at the operator level, but does not directly predict downstream generation quality or uniquely determine the optimal rank. To examine the empirical connection, we compare the operator-fidelity results in Table 3, a new rank-wise evaluation using the Align rel- metric defined in Table 6, and the downstream SC results in Table 5. We use two low-cost operator-level proxies, each computable with a forward evaluation on a small number of batches:
(1) Operator fidelity, computed as the average of the temporal- and height-slice centered cosine similarities reported in Table 3; and
(2) Align rel-, using the metric defined in Table 6,
Thanks for the detailed review. My concerns about the theoretical analysis and the hardware reliance are partially solved. But my concern on more quantitative evaluation including human study remains.
Thank you for further clarifying your remaining concern. We note that newly organizing and collecting a direct human evaluation is not generally included in the standard experimental coverage of closely related efficient-attention studies. Instead, works in this area, including SLA, commonly use VisionReward as a learned proxy for human preference in preference-oriented evaluation. VisionReward is not a direct human study, and its absolute values should be interpreted as relative reward scores rather than probabilities or percentages. Nevertheless, it provides complementary preference-oriented evidence beyond VBench, PSNR, SSIM, and LPIPS.
To provide additional quantitative evidence related to human preference, we evaluated all methods using VisionReward under matched prompts, random seeds, and generation settings:
| Method | VisionReward |
|---|---|
| Dense | 0.1091 |
| VSA | -0.0152 |
| SLA | 0.0874 |
| RoLA | 0.1042 |
RoLA achieves a VisionReward score of 0.1042, only 0.0049 below Dense and 0.0168 above SLA. This result further indicates that RoLA better preserves the perceptual behavior of the dense model than the compared sparse-attention baselines.
We hope this additional result further addresses the reviewer’s concern. We would be happy to discuss any remaining questions or concerns.
We sincerely thank the reviewer for their insightful feedback and constructive comments. Due to the response-length limit, we divide our reply into two parts. This first response addresses the evaluation scope and latency measurement scope. The follow-up response discusses the connection between theory and empirical results, as well as quality preservation in difficult cases.
1. Evaluation Scope (W1)
In addition to the Wan2.1-1.3B and 14B models reported in the main text, the appendix already includes results on Wan2.2-A14B-T2V (Tables 7 and 8). This series covers the most mainstream open-source video diffusion models, spanning different scales, resolutions, frame counts, and both dense and MoE architectures. This evaluation scope is consistent with recent efficient-attention works: SLA and RoPeSLR also primarily evaluate on Wan-series models under standard sequence lengths. Further generalization to longer sequences or alternative RoPE layouts is valuable and left for future work; the current evaluation already provides a reasonably comprehensive picture on representative backbones.
2. Latency Measurement Scope (W2 & Q1)
We thank the reviewer for identifying this ambiguity. The reported 2.63$\times$ speedup is computed from the complete DiT denoising latency in Table 4 (1075.7 s versus 409.3 s), measured under the same NVIDIA H100 configuration. It excludes text encoding and VAE decoding and therefore should not be interpreted as a full text-to-video pipeline speedup. We have corrected the “end-to-end” wording in the revised abstract (p. 1) to “a 2.63$\times$ speedup in DiT denoising.”
We have not measured the complete text-to-video pipeline, including text encoding and VAE decoding, under a single matched hardware configuration, and therefore do not report or claim a full-pipeline acceleration factor. Since RoLA modifies only the DiT self-attention modules, the text encoder and VAE remain unchanged; including these stages would reduce the application-level speedup relative to the reported DiT-denoising speedup.
We also clarify that Figure 1 and Table 4 present separate measurements. Figure 1 reports deployment-oriented results on an RTX 5090, whereas the 2.63$\times$ result is derived exclusively from the matched H100 measurements in Table 4. Results from the two hardware settings are not combined. We retain DiT denoising latency as the primary efficiency metric because it isolates the component modified by RoLA, while explicitly stating its measurement scope.
We discuss the relationship between operator fidelity and downstream quality, as well as difficult-case evaluation, in the follow-up response.