acceptodds
Under review as a conference paper at ICLR 2027

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention incurs quadratic complexity with sequence length, becoming the primary inference bottleneck as video resolution and frame count increase. Sparse low-rank hybrids address this by combining a local sparse branch with a global compressed branch. In video DiTs equipped with 3D Rotary Position Embeddings (RoPE), the global branch faces a structural compatibility issue: when RoPE is applied before a nonlinear feature map, the rotation and nonlinearity generally do not commute, making it difficult to keep a query-independent linear summary while preserving relative rotary geometry. Existing work often sidesteps this issue by replacing genuine cross-token global aggregation with coordinate-conditioned surrogates or learnable absolute positional modules. These compromises can be effective, but they approximate relative decay from absolute coordinates and introduce extra positional parameters. We propose RoLA, a rotary-positioned low-rank linear-attention branch that keeps genuine cross-token aggregation while remaining compatible with a reusable linear summary. The design applies RoPE outside the nonlinear low-rank feature map and reuses a truncated subset of the pre-trained rotary schedule matched to the low-rank bottleneck. This yields a linear-time low-rank global branch with relative positional behavior by design and no additional positional parameters; the full sparse–low-rank module still includes the fixed-sparsity sparse branch. Experiments on open-source video DiTs show that the resulting method remains competitive in generation quality at 90% sparsity while achieving 2.63 end-to-end inference speedup over dense attention on Wan2.1-14B (720p, 81 frames, measured on an NVIDIA H100 GPU).

Primary area:
generative models
Keywords:
Sparse attention, rope, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wJAHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the compatibility problem between 3D RoPE and a query independent low rank global branch in video Diffusion Transformers. It proposes RoLA, which applies a truncated pretrained rotary embedding after a nonlinear low rank feature map, thereby preserving genuine cross token aggregation and a reusable linear summary without additional positional parameters. Mechanism level analyses and generation experiments on Wan2.1 1.3B and 14B show that RoLA remains competitive at 90% sparsity and achieves up to 2.63 times end to end inference speedup over dense attention on an H100 GPU for 720p, 81 frame videos.

Critical strengths
  1. The paper identifies a clear structural tension between 3D RoPE and query independent linear attention in video DiTs. RoLA addresses this tension by applying truncated RoPE after the nonlinear low rank map, preserving relative position structure and a reusable global summary without additional positional parameters.
  2. The method is supported by mechanism level analyses using real Wan2.1 features and controlled positional tests. The results show that applying RoPE outside the activation preserves both position awareness and shift invariance, while the rank sweep supports the selected bottleneck dimension.
  3. RoLA achieves a strong quality and efficiency trade off at 90% sparsity on both Wan2.1 1.3B and 14B. It remains competitive with dense and sparse baselines while achieving a 2.63 times end to end speedup over dense attention on an H100 GPU for 14B 720p videos.
  4. The empirical study includes matched comparisons with VSA, SLA, SALAD, and RoPeSLR, together with controlled ablations of the global branch, rotary rank, activation function, and fine tuning protocol. The detailed algorithm, hyperparameters, complexity analysis, and reproducibility statement make the evaluation relatively transparent.
Critical weaknesses
  1. The evaluation is confined to Wan2.1 1.3B and 14B under standard sequence lengths. Generalization to other video diffusion backbones, longer sequences, alternative RoPE layouts, and broader data distributions remains unestablished.

  2. The primary latency measurements report DiT denoising time and exclude text encoding and VAE stages, while the deployment results in Figure 1 use an RTX 5090 and the main benchmark uses an H100 GPU. The reported end to end acceleration is therefore hardware and pipeline specific, which limits its interpretation as a full application level speedup.

  3. The theoretical results characterize positional behavior and rank truncation under controlled inputs, but they do not connect operator fidelity to generation quality. The theoretical justification for the selected rank and its downstream benefits therefore remains largely empirical.

  4. The quality assessment relies mainly on VBench dimensions and similarity metrics against dense attention outputs. Without human preference studies or systematic failure analysis, the effects on prompt adherence, fine grained spatial relations, and temporal consistency in difficult cases remain unclear.

Questions for authors
  1. The reported 2.63 times end to end speedup appears to measure DiT denoising only, and Figure 1 and the main benchmark use different GPUs. What is the speedup when measured over the complete text to video pipeline, including text encoding and VAE decoding, on a single hardware configuration?
  2. The rank sweep motivates the bottleneck dimension empirically, while the theory section analyzes positional behavior under controlled inputs. Is there any measurable link between operator level fidelity and downstream generation quality that would make the rank choice predictable rather than searched?
  3. VBench and similarity to dense outputs may miss degradations in prompt adherence, fine grained spatial relations, or temporal consistency at 90 percent sparsity. Can additional perceptual evidence be reported for these difficult cases, for example a small human preference comparison or a breakdown on motion heavy prompts?
4 comments

We sincerely thank the reviewer for their insightful feedback and constructive comments. Due to the response-length limit, we divide our reply into two parts. This first response addresses the evaluation scope and latency measurement scope. The follow-up response discusses the connection between theory and empirical results, as well as quality preservation in difficult cases.

1. Evaluation Scope (W1)

In addition to the Wan2.1-1.3B and 14B models reported in the main text, the appendix already includes results on Wan2.2-A14B-T2V (Tables 7 and 8). This series covers the most mainstream open-source video diffusion models, spanning different scales, resolutions, frame counts, and both dense and MoE architectures. This evaluation scope is consistent with recent efficient-attention works: SLA and RoPeSLR also primarily evaluate on Wan-series models under standard sequence lengths. Further generalization to longer sequences or alternative RoPE layouts is valuable and left for future work; the current evaluation already provides a reasonably comprehensive picture on representative backbones.

2. Latency Measurement Scope (W2 & Q1)

We thank the reviewer for identifying this ambiguity. The reported 2.63$\times$ speedup is computed from the complete DiT denoising latency in Table 4 (1075.7 s versus 409.3 s), measured under the same NVIDIA H100 configuration. It excludes text encoding and VAE decoding and therefore should not be interpreted as a full text-to-video pipeline speedup. We have corrected the “end-to-end” wording in the revised abstract (p. 1) to “a 2.63$\times$ speedup in DiT denoising.”

We have not measured the complete text-to-video pipeline, including text encoding and VAE decoding, under a single matched hardware configuration, and therefore do not report or claim a full-pipeline acceleration factor. Since RoLA modifies only the DiT self-attention modules, the text encoder and VAE remain unchanged; including these stages would reduce the application-level speedup relative to the reported DiT-denoising speedup.

We also clarify that Figure 1 and Table 4 present separate measurements. Figure 1 reports deployment-oriented results on an RTX 5090, whereas the 2.63$\times$ result is derived exclusively from the matched H100 measurements in Table 4. Results from the two hardware settings are not combined. We retain DiT denoising latency as the primary efficiency metric because it isolates the component modified by RoLA, while explicitly stating its measurement scope.

We discuss the relationship between operator fidelity and downstream quality, as well as difficult-case evaluation, in the follow-up response.

This response continues our preceding reply and addresses the connection between theory and empirical results (W3 & Q2), followed by quality preservation and difficult-case evaluation (W4 & Q3).

3. Connection between Theory and Empirical Results (W3 & Q2)

We acknowledge that the manuscript presents the operator-level rank sweep and downstream rank ablation separately, without clearly explaining their empirical relationship.

Proposition 2 provides structural motivation by relating the truncation error, under its idealized assumptions, to the energy in the discarded rotary subspace. It therefore motivates rank truncation at the operator level, but does not directly predict downstream generation quality or uniquely determine the optimal rank. To examine the empirical connection, we compare the operator-fidelity results in Table 3, a new rank-wise evaluation using the Align rel-L2L_2 metric defined in Table 6, and the downstream SC results in Table 5. We use two low-cost operator-level proxies, each computable with a forward evaluation on a small number of batches:

(1) Operator fidelity, computed as the average of the temporal- and height-slice centered cosine similarities reported in Table 3; and
(2) Align rel-L2L_2, using the metric defined in Table 6,

{\lVert O_{\mathrm{full}}\rVert_2},$$ newly evaluated on each of the five trained rank variants from Table 5. | Rank $r$ | Operator fidelity $\uparrow$ | Align rel-$L_2$ $\downarrow$ | SC $\uparrow$ | |:--------:|:----------------------------:|:-----------------------------:|:-------------:| | 8 | 0.37 | 0.589 | 0.8788 | | 16 | 0.42 | 0.561 | 0.8910 | | 32 | 0.70 | 0.538 | 0.9398 | | **64** | **0.93** | **0.521** | **0.9521** | | 128 | 1.00 | 0.519 | 0.9540 | The three measurements exhibit a consistent monotonic trend, while the alignment and downstream-quality gains largely saturate at $r=64$. From $r=64$ to $r=128$, Align rel-$L_2$ improves by only 0.002 and SC improves by only 0.0019, while the leading low-rank computation approximately doubles. We therefore select $r=64$ as the smallest evaluated rank near the empirical saturation region. We emphasize that Proposition 2 supplies structural motivation rather than a theoretical prediction of generation quality. The table establishes an empirical association between operator fidelity, output alignment, and downstream quality across the evaluated trained rank variants. These inexpensive proxies may help screen for the saturation region, but downstream validation is still required. This relationship is currently established on Wan2.1 at 90% sparsity; its transfer to other backbones, sparsity levels, sequence lengths, and RoPE layouts remains to be validated. **4. Quality Preservation and Difficult Cases (W4 & Q3)** We clarify that RoLA is designed primarily to preserve the capabilities of the pretrained dense backbone under aggressive attention sparsification, rather than to expand the backbone’s capability boundary. The absolute ability to handle difficult prompts, fine-grained spatial relations, and complex motion is therefore determined to a substantial extent by the underlying dense model. From this perspective, the central evaluation question is whether replacing dense attention with RoLA introduces additional degradation relative to the matched dense baseline. Our evaluation addresses both absolute generation quality and preservation relative to the dense model. The reported VBench and VBench-2.0 dimensions assess absolute generation quality, while PSNR, SSIM, and LPIPS are computed against dense-attention outputs under matched prompts and seeds. In particular, the VBench-2.0 categories of Controllability, Human Fidelity, and Physics provide complementary evidence regarding prompt adherence, human details, and physical plausibility. Taken together, the current results do not indicate a substantial systematic degradation after applying RoLA at 90% sparsity and show that RoLA broadly preserves the behavior of the dense backbone within the evaluated prompts and metrics. We sincerely thank the reviewer once again for their thoughtful comments. We hope the above responses adequately address their concerns and are happy to provide further clarification if needed.

Thanks for the detailed review. My concerns about the theoretical analysis and the hardware reliance are partially solved. But my concern on more quantitative evaluation including human study remains.

Thank you for further clarifying your remaining concern. We note that newly organizing and collecting a direct human evaluation is not generally included in the standard experimental coverage of closely related efficient-attention studies. Instead, works in this area, including SLA, commonly use VisionReward as a learned proxy for human preference in preference-oriented evaluation. VisionReward is not a direct human study, and its absolute values should be interpreted as relative reward scores rather than probabilities or percentages. Nevertheless, it provides complementary preference-oriented evidence beyond VBench, PSNR, SSIM, and LPIPS.

To provide additional quantitative evidence related to human preference, we evaluated all methods using VisionReward under matched prompts, random seeds, and generation settings:

MethodVisionReward ↑\uparrow
Dense0.1091
VSA-0.0152
SLA0.0874
RoLA0.1042

RoLA achieves a VisionReward score of 0.1042, only 0.0049 below Dense and 0.0168 above SLA. This result further indicates that RoLA better preserves the perceptual behavior of the dense model than the compared sparse-attention baselines.

We hope this additional result further addresses the reviewer’s concern. We would be happy to discuss any remaining questions or concerns.

Reviewer_naDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoLA proposes a method use both sparse attention and low-rank linear attention to replace a O(N^2) softmax attention. It puts RoPE after non-linear projection layers which rotates the resulting features using a truncated pretrained rotary schedule. The model is adapted by full flow-matching fine-tuning. Generation experiments cover Wan2.1 and, in the appendix, Wan2.2. For Wan2.1-14B. Finally, RoLA accelerates baseline softmax attention from 1075s to 409s, nearly a 2.6x speedup using a 90% sparsity.

Critical strengths
  1. The paper provides useful controlled evidence beyond a dense-versus-sparse headline. Table 10 fixes the sparse engine and compares no global branch, no RoPE, inside-activation RoPE, and outside-activation RoPE , while Tables 5 and 11 study rank and sparsity. These controls support the practical benefit of the proposed branch in the reported setting
  2. The construction has a clear computational interpretation and encouraging reported performance across model scales. Table 4 reports lower latency than the sparse-global baselines and relatively preserved selected VBench dimensions on both Wan2.1 model sizes; Appendix D adds Wan2.2 and broader VBench-2.0 evaluation.
Critical weaknesses
  1. The distinctive rotary-truncation design is insufficiently justified. Post-feature-map rotation already appears in RoFormer. Under the official Wan layout (44,42,42), if the first 64 real rotary coordinates are retained in the official Wan ordering, the allocation would be (44,20,0), leaving no explicit width-axis rotation in the global branch. Please clarify whether the implementation follows this ordering.
  2. The quality and speed advantages over SLA need clearer evidence. The reported 1.44–1.47× DiT speedups lack a timing breakdown separating branch design from routing and kernel optimizations. Offloading descriptions conflict, and the advertised end-to-end speedup excludes text encoding and VAE. Qualitative frames appear distorted in aspect ratio, several scenes are dark, and clearly assessable human details and motion are limited, making the visual advantage difficult to judge.
  3. The architecture, equations, and complexity accounting are mostly clear, and the paper distinguishes the linear branch from the full module's fixed-sparsity quadratic cost. However, important descriptions conflict: the stated Wan rotary allocation/native-prefix truncation is inconsistent with the official implementation; Appendix E.1 says offloading is eliminated while Table 12 explicitly uses offloading; and the abstract calls the 2.63× result end-to-end although the experimental protocol excludes text encoding and VAE. These discrepancies affect the reader's understanding of what was actually implemented and measured.
Questions for authors
  1. Please provide the exact rotary-coordinate indices and axis allocation used for rank 64, with the corresponding code. If native-prefix selection is used, how does the global positional kernel behave for a pure width displacement with content held fixed? If coordinates are interleaved or reallocated, how should the native-ordering description and Figure 4 be interpreted?
  2. What accounts for the approximately 1.44×/1.47× DiT speedups over SLA in Table 4? Please provide a matched latency breakdown for routing/mask construction, sparse attention, global-branch projections and aggregation/readout, normalization/gating/fusion, and the remaining DiT computation. For fused kernels, report their combined timing and state which operations are included. And also for hardware, precision, sequence length, denoising steps, and specify the timing protocol.
  3. Beyond RoFormer's established post-feature-map rotation, which measured gains depend specifically on retaining a native rank-64 prefix?
4 comments

We sincerely thank the reviewer for their insightful feedback and constructive comments. Due to the response-length limit, we divide our reply into two parts. This first response addresses the rotary-truncation design and its relationship to RoFormer. The follow-up response addresses the measurement discrepancies, latency breakdown, and qualitative comparison.

1. Rotary Truncation Design (W1 & Q1)

We explicitly adopt native-prefix truncation according to the pretrained backbone’s original coordinate ordering: we retain the first rr rotary coordinates and do not reorder them by frequency or reallocate them across axes. Under the official Wan layout (dt,dh,dw)=(44,42,42)(d_t,d_h,d_w)=(44,42,42), retaining the first 64 coordinates yields the exact allocation (44,20,0)(44,20,0): the full temporal allocation, a partial height allocation, and no width-axis coordinates.

We acknowledge that the manuscript’s description of the official layout and the resulting axis allocation was incorrect. We have corrected Section 3.2 (p. 5) to report the official layout and the exact rank-6464 allocation, and have clarified that no frequency reordering or cross-axis reallocation is performed.

Consequently, with token content held fixed, a pure width displacement does not change the positional component of the rank-6464 global kernel, because no width-axis coordinates are retained. This applies only to the low-rank global branch; the sparse branch continues to use the full pretrained 3D-RoPE schedule and therefore retains width-relative positional information. Figure 4 should accordingly be interpreted as evaluating truncation fidelity on the retained temporal and height components, rather than as a width-axis fidelity evaluation at r=64r=64.

The low-rank branch is positioned as a compensator for the smooth global background discarded by the sparse branch; fine-grained per-axis relative structure is primarily handled by the sparse branch. Consequently, a reduced rotary budget on certain axes, especially the width axis, is intentional rather than an omission. Existing experiments support the practicality of this choice from both operator fidelity and downstream quality. Figure 4 and Table 3 show that, as the retained rank increases from 8 to 64, centered cosine similarity and normalized L2L_2 error improve monotonically and saturate on both temporal and height slices. Table 5 further shows that downstream VBench metrics already obtain clear gains at r=64r=64, with only marginal changes when increasing to 128. Because the low-rank branch plays a compensatory rather than fine-grained local modeling role, reducing the rotary dimension on certain axes does not significantly impair overall performance.

2. Relationship to RoFormer and the Contribution of the Native Rank-64 Prefix (Q3)

Post-feature-map rotation already appears in RoFormer, yet RoLA addresses the more constrained setting of sparse–low-rank hybrids in pretrained 3D-RoPE video DiTs. In this regime, the low-rank global branch must simultaneously maintain a query-independent linear summary, preserve relative-position interactions under the pretrained rotary geometry, introduce no extra positional parameters, and remain compatible with the existing sparse engine. When rotation is applied before the pointwise nonlinearity, rotation and nonlinearity generally do not commute; the resulting key-side summary can still be linearly aggregated, but it becomes entangled with absolute positions and cannot cleanly admit a relative-offset factorization (see Proposition 3 and the mechanism experiments in Section 4.1, Figure 5, and Table 2).

RoLA’s design—applying rank-truncated RoPE after the low-rank nonlinearity and directly reusing a truncated subset of the pretrained rotary schedule—is precisely what enables these constraints to be satisfied simultaneously. Compared with simply adapting the corresponding design from RoFormer to the same sparse backbone, the additional benefits of RoLA lie mainly in satisfying these constraints at once. The mechanism experiments in Table 2 and Figure 5, together with the controlled global-branch ablation in Appendix Table 10, provide evidence for the corresponding components of the design.

As for the native rank-6464 prefix itself, we regard it as a simple, parameter-free choice that is fully compatible with the pretrained backbone. Under this choice, both operator fidelity and downstream metrics already saturate, indicating that the retained subspace is sufficient for the smooth global background that the low-rank branch is intended to model. We do not claim that the native prefix is uniquely optimal among all possible ways of selecting 64 coordinates; its sufficiency, rather than optimality, is the key empirical observation.

We address the measurement discrepancies, latency breakdown, and qualitative comparison in the follow-up response.

This response continues our preceding reply and addresses the measurement discrepancies, latency breakdown, and qualitative comparison in W2, W3, and Q2.

3. Clarification of Conflicting Descriptions (W3)

We note that the descriptions of CPU offloading in Appendix E.1 and Table 12 were inconsistent because the two deployment settings were not clearly distinguished. Under CPU block offloading, RoLA reduces peak GPU memory from 20.43 GiB to 9.41 GiB compared with Dense, achieving a 2.2$\times$ reduction. Separately, RoLA can run without CPU offloading, with a measured peak GPU memory of 30.55 GiB. We have revised Appendix E.1 (p. 19) and Table 12 (p. 21) to clearly distinguish these two settings; Table 12 now includes a separate no-offloading row for RoLA.

We note that reporting the complete DiT denoising time and speedup is common practice in this line of work. Specifically, our reported 2.63$\times$ speedup measures the complete DiT denoising process under matched experimental settings, excluding text encoding and VAE operations, rather than the latency of the complete video-generation pipeline. We thank the reviewer for identifying this issue. We agree that the “end-to-end” claim in the abstract is inaccurate and have corrected it in the revised abstract (p. 1) to “a 2.63$\times$ speedup in DiT denoising.”

4. Latency Breakdown of the Speedup over SLA (W2 & Q2)

To isolate the sources of the reported DiT speedup, we performed a matched component-wise latency breakdown under identical conditions for both methods:

  • Precision: BF16 for all attention and remaining modules.
  • Sparsity: 90% (topk = 0.1) with the same block size (BLKQ = BLKK = 64).
  • Sparse engine: the official SLA kernel versus the RoLA sparse kernel used in Table 4.
  • Input setting: 480p for Wan2.1-1.3B and 720p, 81 frames for Wan2.1-14B.
  • Denoising steps: 50, following the default Wan inference setting and matching Table 4.
  • No torch.compile or CUDA Graph.
  • Hardware: NVIDIA H100 80GB GPU.

Timing was measured with torch.cuda.Event using 2 warm-up runs, followed by an average over 20 runs. The breakdown is shown below (unit: ms, mean ± std):

ComponentSLA (1.3B)RoLA (1.3B)SLA (14B)RoLA (14B)
Routing / Mask construction691.98 ± 0.12489.97 ± 0.5110881.62 ± 30.368315.41 ± 5.96
Sparse attention19524.63 ± 5.199288.09 ± 7.96422233.95 ± 131.21213581.19 ± 34.87
Global-branch projections–173.26 ± 0.07–1705.99 ± 0.31
Global-branch agg. / readout–146.86 ± 0.25–1242.39 ± 0.08
Norm / Gate / Fusion1.15 ± 0.00170.89 ± 0.051.58 ± 0.001710.56 ± 0.38
Self-attn I/O (QKV + out proj)1419.31 ± 0.291571.04 ± 1.3343527.74 ± 87.0145662.00 ± 25.71
Self-attn subtotal21643.97 ± 5.3511849.31 ± 9.49476654.16 ± 50.96272229.92 ± 33.27
Remaining DiT9453.499832.06132947.05136404.30
Total DiT denoising31097.46 ± 5.8421681.37 ± 15.50609601.22 ± 12.72408634.22 ± 44.64
Speedup1.0$\times$1.43$\times$1.0$\times$1.49$\times$

The observed speedup—1.43$\times$ on 1.3B and 1.49$\times$ on 14B—is driven primarily by the sparse-attention kernel and reduced routing cost. On the 14B model, sparse-attention time decreases from 422.23 s to 213.58 s, while the additional global-branch overhead, including projections, aggregation/readout, and normalization/gating/fusion, remains small at approximately 1.1% of the total DiT denoising time. On the 1.3B model, the relative global-branch overhead is approximately 2.3%, and self-attention constitutes a smaller fraction of the total runtime, resulting in a somewhat smaller overall speedup.

RoLA’s fused Triton kernels combine several operations; their internal costs cannot be further subdivided and are reported under the corresponding rows above. The two global-branch matrix multiplications—context accumulation and readout—are timed separately outside the fused kernels.

5. Qualitative Comparison (W2)

The purpose of our qualitative comparison is primarily to assess whether RoLA preserves the behavior of the matched dense model, rather than to claim that the attention replacement independently improves the backbone’s absolute generation capability. We acknowledge that the current selection and presentation, including the displayed aspect ratios and several dark scenes, may make this relative comparison difficult to assess.

We sincerely thank the reviewer once again for their thoughtful comments. We hope the above responses adequately address their concerns and are happy to provide further clarification if needed.

Thanks author for the reply. I think I got my answers. These address my concerns about the rotary allocation, offloading, and timing scope. However, I remain unconvinced that the methodological novelty and evidence of visual quality advantages over SLA are sufficient for acceptance. I therefore maintain my rating.

Thank you for your careful review. We would like to further clarify RoLA’s contribution relative to SLA. SLA primarily relies on a sparse local branch, whereas RoLA additionally introduces a query-independent low-rank global branch under the same sparsity setting. This branch applies RoPE after the nonlinear feature mapping, reuses a rank-truncated subset of the pretrained 3D-RoPE schedule, and is fused with the sparse-branch output through token-wise gating. Therefore, RoLA is not merely a different sparse kernel or routing implementation. Its core contribution is a rotary-positioned global compensation mechanism that preserves and models relative positional information—an aspect not provided by prior approaches to our knowledge—to recover long-range information discarded by sparse attention.

The controlled ablations support these design contributions. Table 10 evaluates the effect of adding and modifying the global branch under a fixed sparse engine, while Table 2 and Figure 5 isolate the relative-position behavior enabled by applying rotation outside the activation function. The rank sweep further shows that the global branch remains effective with a rank-6464 rotary bottleneck.

Under the same 90% sparsity setting used in Table 4, RoLA also achieves a measured quality–efficiency improvement over SLA. On Wan2.1-1.3B, RoLA reduces DiT latency from 31.2 s to 21.7 s, while improving SC from 0.9401 to 0.9521, MS from 0.9782 to 0.9812, IQ from 0.6343 to 0.6492, and AQ from 0.5918 to 0.6068. On Wan2.1-14B, it reduces latency from 601.0 s to 409.3 s, while improving SC from 0.9481 to 0.9629, MS from 0.9802 to 0.9853, IQ from 0.6398 to 0.6783, and AQ from 0.6052 to 0.6294. These results indicate that the rotary-positioned global branch effectively compensates for information lost through sparse attention while retaining the efficiency benefits of the sparse engine.

If you have any further concerns, we would be happy to discuss them further.

Reviewer_gqsQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed RoLA pairs sparse attention with a rotary low-rank global branch for pretrained video diffusion Transformers. The global branch projects queries and keys to low dimensions with SiLU before RoPE, using a truncated RoPE frequency schedule. Key-value summaries, RMS norm, and gated fusion recover global information lost to sparsity. On the Wan models, RoLA beats sparse attention baselines, and at 90% sparsity it speeds up Wan2.1-14B denoising by 2.63×. Note that linear complexity holds only for the global branch; overall complexity stays quadratic at a fixed sparsity ratio.

The efficiency gains and ablation results are promising, but the contribution over existing Rotary Linear Attention methods is not clearly established. The rotation frequency allocation also needs more theoretical or experimental support, and some implementation details differ between the main text and supplementary material, which hurts reproducibility.

Critical strengths

1.Coherent global information compensation. RoLA applies rotary position encoding after nonlinear feature mapping. This preserves relative positional structure in pairwise attention and allows reusable global key-value summaries. RMS normalization and gated fusion then integrate signed rotary features with the sparse attention branch.

2.Useful controlled experiments and practical potential. Table 10 compares global branch designs with the sparse attention engine and routing fixed. Table 9 controls for dense-model fine-tuning. Along with the reported denoising speedup, these results show RoLA's practical potential. However, it does not fully match the original model's generation quality.

Critical weaknesses
  1. Discrepancies between the manuscript and implementation. Sections 3.1 and 3.4 differ from the supplementary code in projection sharing and gating design. Training hyperparameters and sparsity levels (90% vs. 95%) are also inconsistent. Furthermore, Section E.1 contradicts Table 12 regarding CPU offloading. These discrepancies require clarification to ensure reproducibility, particularly for the reported 2.63× speedup.

  2. Insufficient validation of novelty and rotary truncation. RoFormer already combines nonlinear mappings with rotary encoding, leaving RoLA’s novelty insufficiently demonstrated. Under Wan2.1’s RoPE ordering, retaining the first 64 coordinates preserves temporal and height information but excludes width information, contradicting Section 3.2. Controlled comparisons and axis-specific ablations are needed.

Questions for authors
  1. Were Tables 4 and 10 generated using head-specific or shared query/key projections? Was the gating mechanism token-wise or token-and-head-wise? Please reconcile the manuscript with the supplementary code and clarify the exact experimental configurations.

  2. Which rotary frequencies are retained at (r=64)? Under the described native-prefix truncation, no width-axis frequencies appear to be preserved. Please clarify this allocation, reconcile it with Section 3.2, and justify the choice empirically.

  3. What additional benefits does RoLA provide compared with adapting RoFormer's Eq. (19) to the same sparse Wan backbone? Which existing experimental results isolate the contributions of RoLA's additional design choices?

3 comments

We sincerely thank the reviewer for their insightful feedback and constructive comments. We address each point below.

1. Code and Experimental Configurations (W1 & Q1)

We provided a simplified demonstration because the original implementation contains credited custom operators that cannot be disclosed under the double-blind policy. Upon examination, we confirm that the demonstration differs from the manuscript and was not used to produce the reported results; we will revise it accordingly. Our actual implementation follows the manuscript: Tables 4 and 10 use head-specific query/key projections and token-wise gating.

We also acknowledge that the CPU-offloading descriptions were unclear and have clarified them in the revised Appendix E.1 (p. 19) and Table 12 (p. 21). The training hyperparameters and sparsity settings will likewise be made consistent across the manuscript and implementation to ensure reproducibility.

2. Novelty Relative to RoFormer (W2 & Q3)

RoFormer primarily introduces rotary relative-position encoding into attention for causal autoregressive language models, where positional information supports sequential next-token prediction. In contrast, video DiTs perform bidirectional generation over the entire spatiotemporal token volume and must preserve relative geometry across all query-key pairs to maintain semantic consistency, motion coherence, and stable scene layout.

In this setting, the low-rank global branch must simultaneously (i) maintain a query-independent linear summary for linear-complexity, once-compute-many-reuse execution, (ii) preserve relative-position interactions under the pretrained 3D rotary geometry, (iii) introduce no additional positional parameters, and (iv) remain compatible with the sparse engine.

When rotation is applied before the pointwise nonlinearity, rotation and activation generally do not commute. Although the key-side summary remains linearly aggregatable, it becomes entangled with absolute positions and cannot cleanly admit a reusable relative-offset factorization (Proposition 3; Section 4.1; Figure 5; Table 2). This is particularly important because the global summary is shared by every query.

RoLA addresses this problem by applying rank-truncated RoPE after the low-rank nonlinearity and reusing a truncated subset of the pretrained rotary schedule. This preserves relative rotary structure within a reusable query-independent summary without introducing additional positional parameters, unlike approaches that avoid genuine cross-token aggregation or introduce auxiliary absolute-position modules.

RoLA uses the global branch as an unnormalized, gated residual correction to the sparse output rather than as a standalone attention distribution. Its magnitude is controlled by RMSNorm and the token-wise gate, consistent with its residual-compensation role.

Compared with directly adapting RoFormer’s Eq. (19) to the same sparse Wan backbone, RoLA combines post-activation rotary encoding, a reusable low-rank global summary, truncated reuse of pretrained 3D RoPE, and gated sparse–global fusion. Table 2 and Figure 5 evaluate rotation placement and relative-position behavior, while Appendix Table 10 evaluates the global branch under a fixed sparse engine.

3. Rotary Truncation (W2 & Q2)

Under Wan2.1’s native 3D-RoPE ordering, the head dimension dh=128d_h=128 is partitioned as (dt,dh,dw)=(44,42,42)(d_t,d_h,d_w)=(44,42,42). Section 3.2 incorrectly states (42,43,43)(42,43,43) and consequently describes r=64r=64 as retaining information from both spatial axes.

Under the official layout and native-prefix truncation, retaining the first 64 coordinates yields the exact allocation (44,20,0)(44,20,0): all 44 temporal coordinates, the first 20 height coordinates, and no width coordinates. We thank the reviewer for identifying this error. Section 3.2 (p. 5) has been corrected to explicitly report the retained indices and axis allocation.

The low-rank branch models the smooth global background discarded by the sparse branch, while fine-grained axis-specific structure is primarily handled by the sparse branch. Figure 4 and Table 3 show that centered cosine similarity and normalized L2L_2 error improve monotonically and largely saturate by r=64r=64 on the evaluated temporal and height slices. Table 5 shows the same downstream trend, with substantial VBench gains up to r=64r=64 and only marginal changes at 128. These results support the sufficiency of r=64r=64 for the complete sparse–low-rank architecture.

Reviewer_gqsQOpenReview ↗

Response to the authors' reubttal

Thank you for clarifying the implementation used in the experiments and the rank-64 rotary allocation. I still have a reproducibility concern, because the original supplementary code differs from the implementation used to report the results. The response to Q3 clarified how RoLA differs from RoFormer, but the reported comparison still retains what was added by adapting Equation (19) onto the same sparse WAN backbone. I maintain my original rating.

Dear Reviewer, Thank you for your further feedback. We have now directly revised the code in the supplementary material so that it matches the implementation used for the experiments reported in the paper, rather than merely explaining the discrepancies in our response. The revised implementation uses independent query/key projections for each attention head, a token-wise scalar gate shared across heads in both the reference and fused implementations, and 90% sparsity as the default setting for the main experiments. We have also corrected the training configuration, package interfaces, and weight layouts in the fused kernels, and added explicit implementation-to-paper correspondence and revision notes.

Regarding Eq. (19), we would like to define the scope of our contribution more precisely. The operation of first applying the feature map and then RoPE in the numerator of linear attention is inherited from RoFormer, and we do not claim that this operation alone is novel. A direct adaptation of Eq. (19) retains the rotary-positioned numerator while using an unrotated linear-attention denominator. RoLA instead treats the rotary-positioned global branch as an unnormalized residual correction to sparse attention, using an RMS-normalized readout together with token-wise gated fusion that accounts for the scale of the sparse branch.

Because the remaining rebuttal period is insufficient to post-train and comprehensively evaluate an additional variant under the same data and optimization budget, we designed a controlled frozen-checkpoint operator-level experiment to isolate this architectural difference without introducing variations caused by retraining. Specifically, we hold fixed the trained checkpoint, sparse branch, head-specific rank-6464 projections, rotary-positioning scheme, cached Q/K/VQ/K/V activations, and all learned parameters. We change only the global readout from RoLA’s RMS-normalized residual readout to the denominator-normalized form of Eq. (19), and measure its alignment with both the dense-attention output and the residual defined as dense attention minus sparse attention:

Global readoutFused-output rel-L2L_2 error ↓Residual rel-L2L_2 error ↓Residual cosine similarity ↑
Eq. (19)-style denominator normalization0.6140.9040.50
RoLA RMS readout0.5310.6560.69

This frozen-checkpoint diagnostic shows that, when all learned parameters and input activations are held fixed, RoLA’s residual readout more accurately approximates both the dense-attention output and the global information omitted by the sparse branch. This allows us to isolate the effect of the readout design while retaining the rotary numerator adopted from Eq. (19).

We emphasize that this diagnostic is not intended to replace a comparison between separately post-trained models. Instead, it is designed to isolate the proposed design under the strictest feasible control within the rebuttal period, avoiding the confounding effects of different optimization outcomes on the architectural comparison.

We thank the reviewer for identifying the reproducibility issue and for prompting us to distinguish the components inherited from RoFormer from RoLA’s additional residual-compensation design. We hope these additions further address your concerns and will be taken into consideration in the final assessment.

Comments on the paper

1 comment
Area_Chair_5ryNOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. The rebuttal is now available in the discussion forum. I would like to kindly remind you to check the rebuttal and leave comments for the authors, which would be very helpful.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.