acceptodds
Under review as a conference paper at ICLR 2027

RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

Abstract

Diffusion Transformers (DiTs) have become dominant for high-fidelity video generation, yet their attention cost bottlenecks long-sequence synthesis. Recent sparse-linear hybrids reduce this cost but degrade at extreme sparsity due to what we call the RoPE dilemma: positive feature maps enabling linear attention do not preserve 3D Rotary Position Embeddings (RoPE) orthogonality, losing explicit relative spatiotemporal offsets in global aggregation. We propose RoPeSLR, a 3D RoPE-aware sparse-low-rank framework. Through structural analysis, we observe that DiT attention matrices empirically exhibit decomposition into high-energy sparse spikes and smooth backgrounds with slowly growing rank. Motivated by this observation, we design an energy-driven block-sparse branch for dominant interactions and a Fourier low-rank compensator preserving RoPE structure via rotated Fourier features aggregating global context in . At 90% sparsity, RoPeSLR maintains strong VBench quality on Wan2.1-1.3B and 14B while reducing DiT denoising latency by and , respectively.

Primary area:
generative models
Keywords:
Rope, sparse attention, low rank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hxT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RoPeSLR is a sparse plus low-rank attention for video DiTs. A block-sparse branch computes exact attention on blocks selected from pooled query/key energy, at about 90% sparsity. A low-rank branch applies RoPE rotations to a subset of frequency pairs of q and k, maps them through a learned positive map (linear projection plus sigmoid, r = 64), and aggregates over all tokens, linear-attention style. The outputs are fused with a token-wise gate and a per-head RMSNorm. Training first aligns the new parameters to full attention (MSE loss), then fine-tunes the full model with flow matching. The design is motivated by a "RoPE dilemma" in prior sparse-linear hybrids and by a structural proposition: O(L^{1.5}) sparse non-zeros, rank independent of L, and O(L^{-1/2}) entry-wise error. On Wan2.1-1.3B and 14B at 90% sparsity, the paper reports five VBench dimensions, PSNR/SSIM/LPIPS against full attention, and 1.67x / 1.80x reductions in DiT latency, with slightly better quality than SLA at similar latency.

Critical strengths
  1. Experiments are run at both 1.3B and 14B scale on Wan2.1. At matched 90% sparsity there are consistent, if small, gains over SLA, and a long-sequence latency test on HunyuanVideo is included. Adding a global compensator to block-sparse attention is a sensible direction, and these results suggest it is competitive.

  2. The diagnostics (energy split and stable rank, per-frequency interaction magnitudes, gate maps, SVD of the low-rank branch output) give some intuition about how work is divided between the two branches.

Critical weaknesses
  1. The theory does not support the design and contains errors.
  • With τ=cL−1/2\tau = cL^{-1/2} and E=τ/2E = \tau/2, simply discarding the background already gives a maximum entry error of at most τ=2E\tau = 2E with rank 0. The proposition therefore improves on the trivial bound by at most a factor of 2.
  • Background entries average 1/L1/L, so an O(L−1/2)O(L^{-1/2}) entry-wise tolerance does not control the row-wise error that determines the attention output.
  • The construction (Cor. 1) needs the true partition function and Ωτ\Omega_\tau, i.e. the full L×LL \times L matrix, and R=4(exp⁡(4Qmax⁡2)−1)R = 4(\exp(4Q_{\max}^2) - 1) is vacuous.
  • Lemma 3's rank bound O(dheadlog⁡(1/δ))O(d_{\text{head}} \log(1/\delta)) is weaker than the trivial rank(S)≤dhead\mathrm{rank}(S) \le d_{\text{head}}.
  • Remark 2's "one third" result disappears once the rank is summed over the three axes.
  • Lemma 1 uses dhead\sqrt{d_{\text{head}}} where LayerNorm gives dmodel\sqrt{d_{\text{model}}}, and it ignores adaLN modulation. This matters because the paper presents this analysis as the principled basis that distinguishes it from "heuristic" hybrids.
  1. The central claim, that a RoPE-preserving compensator is what helps, is not isolated, and the gains are confounded by fine-tuning.
  • The deployed map (learned linear projection plus sigmoid on absolutely rotated features) breaks the shift-invariance the paper argues for (Sec. 3.2). Rotating features inside linear attention already appears in RoFormer.
  • Table 3 only compares sparse-only with the full method. There is no same-budget comparison with a standard linear compensator (SLA-style), RoFormer-style RoPE linear attention, all frequency pairs, or no rotation.
  • RoPeSLR beats full attention on SC and MS at 80-90% sparsity (Tables 2 and 4), which suggests that fine-tuning on curated data shifts the metrics. Yet there is no fine-tuned full-attention reference.
  • Only 5 of the 16 VBench dimensions are reported (no Dynamic Degree, no semantic dimensions), with no seeds or variance, and SVG2 runs at 72% / 80% sparsity rather than 90%.
  • Latency against SLA is essentially the same (30.4 vs 31.2 s; 598.3 vs 601.0 s), so the paper stands or falls on this quality comparison.
Questions for authors
  1. Can you report the following variants with the same sparse branch, data and training budget: a standard linear compensator, RoFormer-style RoPE linear attention, all frequency pairs, and no rotation?
  2. Can you add a full-attention model fine-tuned on the same data for the same number of steps? Did SLA and VSA receive the same data and steps?
  3. How do SLA and RoPeSLR compare at 95% and 97% sparsity, given the "extreme sparsity" motivation?
  4. Which frequency pairs are in I^h, and how many? Small m corresponds to the highest rotational frequencies; why are these suited to a smooth global background?
  5. What are the block size and threshold, and how is the 90% ratio enforced? The low-rank branch sums over all tokens, including those already covered by the sparse branch; how is this double counting handled?
Reviewer_SJ2ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoPeSLR, a 3D RoPE-aware sparse-low-rank attention framework for accelerating video diffusion Transformers. The method is motivated by structural analysis and empirical observations that post-softmax attention matrices contain high-energy sparse interactions and low-energy residuals with low stable rank. It combines an energy-driven block-sparse branch, which computes softmax attention over retained blocks, with a Fourier low-rank branch, which selects leading RoPE frequency pairs, applies a learnable positive feature map, and aggregates global context through compact key–value states. The two branch outputs are combined using a token-wise scalar gate followed by RMSNorm. The framework is adapted to pretrained DiTs through two-stage post-training: first, the backbone is frozen while newly introduced parameters are trained to align the fused output with full-attention outputs; subsequently, the backbone is unfrozen for joint fine-tuning. Experiments on Wan2.1-1.3B and Wan2.1-14B report approximately 1.67× and 1.79× speedups, respectively, at 90% attention sparsity, with competitive quality on the reported metrics.

Critical strengths

Empirical structural analysis. The paper examines attention-energy concentration and the stable rank of low-energy residuals, including their behavior across sequence lengths, providing empirical motivation for the sparse–low-rank design.

Measured acceleration. Experiments report approximately 1.67× and 1.79× speedups on Wan2.1-1.3B and Wan2.1-14B at 90% attention sparsity, with competitive results on the reported metrics.

Component ablations. The paper includes ablations on the low-rank compensator, sparsity, feature dimension, and activation function, helping characterize the contribution of the additional branch and several design choices.

Critical weaknesses

The relationship between the reported fidelity metrics and the qualitative examples is insufficiently clear. Table 2 reports PSNR values of approximately 25–26 dB for the fine-tuned models, while Figure 1 shows noticeable visual differences from the full-attention baseline. Based on my experience with these metrics, the qualitative differences are difficult to reconcile with the reported fidelity results. The paper does not adequately explain how the illustrated examples relate to the aggregate scores. Since Stage II updates the pretrained backbone, clearer information about the reference model and evaluation procedure is particularly important for assessing how faithfully the final model preserves the original model’s outputs.

The practical efficiency advantage over training-free approaches is not convincingly established. On Wan2.1-T2V-14B, Table 2 reports only approximately 1.36× speedup for SVG2 at 80% sparsity. This differs from my own measurements, where approximately 70% sparsity provides around 1.6× speedup while retaining dense attention during the first 30% of denoising steps. The SVG2 inference configuration is insufficiently documented to explain this gap, raising concerns about how representative the reported baseline is. RoPeSLR achieves approximately 1.8× speedup at 90% sparsity, but this result alone does not establish a compelling practical benefit from the additional training. Although Table 2 reports quality improvements, a clearer comparison against a well-configured SVG2 baseline is needed to assess the actual quality–latency advantage.

Limited algorithmic novelty beyond existing sparse–linear attention. The central idea of separating sparse high-magnitude interactions from a low-rank remainder has already been articulated in SLA. The sparse branch is described through pooled Q/K scoring and block selection, without clearly establishing an algorithmic distinction from existing block-sparse approaches. The main potential advance therefore lies in the RoPE-aware Fourier compensator and its structural analysis. However, Equation (7) follows from the standard RoPE identity, while the default learned positive feature map explicitly relaxes exact kernel-level shift-invariance. The theoretical positional argument therefore does not directly establish the claimed advantage of the deployed mechanism. Figures 4 and 11 provide useful mechanistic evidence, but the paper lacks a comparison against a generic RoPE-aware linear or low-rank compensator under otherwise matched experimental conditions, as well as final generation-quality results for the variant without the positional prior. Consequently, the current evidence does not convincingly establish a sufficiently substantial advance over existing sparse–linear attention methods.

Questions for authors
  1. PSNR results and qualitative differences.

The paper explicitly states that Stage II fine-tunes the model weights. I would generally expect such fine-tuning to produce noticeable visual differences between videos generated by the fine-tuned model and those generated by the original model, as also illustrated in Figure 1. However, Table 2 reports PSNR values of approximately 25–26 dB. Based on my previous experience, a PSNR of around 26 dB usually corresponds to very small visual differences, largely confined to local details, while the overall scene composition and content remain almost identical. The noticeable differences in Figure 1 therefore leave me confused about how to reconcile the qualitative examples with the reported PSNR values. Could the authors clarify this apparent discrepancy?

2.Latency results and SVG2 inference settings

I have some concerns about the latency results in Table 2. In my own experiments with SVG2 on Wan2.1-T2V-14B, approximately 70% sparsity typically provides around 1.6× speedup over full attention, even while keeping the first 30% of denoising steps dense. However, Table 2 reports only approximately 1.36× speedup for SVG2 at 80% sparsity. Moreover, RoPeSLR achieves only approximately 1.8× speedup despite using a very high sparsity of 90%. My understanding is that most training-free methods can already achieve comparable speedups. Could the authors provide the exact inference parameters used for the SVG2 baseline under the reported 720p, 81-frame setting?

3.What evidence isolates the benefit of the RoPE-aware Fourier construction from that of adding a trainable global compensator?

Table 3 establishes the usefulness of the complete low-rank compensator relative to sparse attention alone. Could the authors additionally compare it with a generic linear or low-rank compensator under otherwise matched experimental conditions? Figure 11 provides a useful representation-spectrum comparison with and without the positional prior; could the corresponding final generation-quality results also be reported? These comparisons would help determine how much of the improvement specifically comes from the proposed RoPE construction.

Reviewer_hy3HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RoPeSLR, a sparse–low-rank attention framework for efficient video Diffusion Transformers. It combines an energy-driven block-sparse attention branch with a RoPE-aware Fourier low-rank compensator to reduce computational costs while maintaining generation quality.

Critical strengths

The structural analysis is meaningful and provides theoretical motivation for decomposing DiT attention into sparse and low-rank components. The experimental results also demonstrate a favorable trade-off between generation quality and inference efficiency.

Critical weaknesses

The preservation of RoPE properties is a central motivation of the proposed method, but it lacks direct validation. Although the paper shows that relative-position dependence is preserved before the learnable feature mapping, it remains unclear whether these properties are retained afterward. The current experiments also do not sufficiently isolate the contribution of RoPE preservation from that of general low-rank compensation.

Questions for authors

Figure 4 shows periodic patterns in output similarity, but these may also arise from the temporal structure of video tokens. How does this experiment distinguish the effects of RoPE preservation from the inherent similarity between temporally related tokens?

Can the authors clarify whether the learned positive feature mapping preserves the relative-position properties of RoPE? Is there evidence that this preservation, rather than generic global low-rank compensation, accounts for the observed improvements in generation quality?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.