- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
RoPeSLR is a sparse plus low-rank attention for video DiTs. A block-sparse branch computes exact attention on blocks selected from pooled query/key energy, at about 90% sparsity. A low-rank branch applies RoPE rotations to a subset of frequency pairs of q and k, maps them through a learned positive map (linear projection plus sigmoid, r = 64), and aggregates over all tokens, linear-attention style. The outputs are fused with a token-wise gate and a per-head RMSNorm. Training first aligns the new parameters to full attention (MSE loss), then fine-tunes the full model with flow matching. The design is motivated by a "RoPE dilemma" in prior sparse-linear hybrids and by a structural proposition: O(L^{1.5}) sparse non-zeros, rank independent of L, and O(L^{-1/2}) entry-wise error. On Wan2.1-1.3B and 14B at 90% sparsity, the paper reports five VBench dimensions, PSNR/SSIM/LPIPS against full attention, and 1.67x / 1.80x reductions in DiT latency, with slightly better quality than SLA at similar latency.
Critical strengths
-
Experiments are run at both 1.3B and 14B scale on Wan2.1. At matched 90% sparsity there are consistent, if small, gains over SLA, and a long-sequence latency test on HunyuanVideo is included. Adding a global compensator to block-sparse attention is a sensible direction, and these results suggest it is competitive.
-
The diagnostics (energy split and stable rank, per-frequency interaction magnitudes, gate maps, SVD of the low-rank branch output) give some intuition about how work is divided between the two branches.
Critical weaknesses
- The theory does not support the design and contains errors.
- With and , simply discarding the background already gives a maximum entry error of at most with rank 0. The proposition therefore improves on the trivial bound by at most a factor of 2.
- Background entries average , so an entry-wise tolerance does not control the row-wise error that determines the attention output.
- The construction (Cor. 1) needs the true partition function and , i.e. the full matrix, and is vacuous.
- Lemma 3's rank bound is weaker than the trivial .
- Remark 2's "one third" result disappears once the rank is summed over the three axes.
- Lemma 1 uses where LayerNorm gives , and it ignores adaLN modulation. This matters because the paper presents this analysis as the principled basis that distinguishes it from "heuristic" hybrids.
- The central claim, that a RoPE-preserving compensator is what helps, is not isolated, and the gains are confounded by fine-tuning.
- The deployed map (learned linear projection plus sigmoid on absolutely rotated features) breaks the shift-invariance the paper argues for (Sec. 3.2). Rotating features inside linear attention already appears in RoFormer.
- Table 3 only compares sparse-only with the full method. There is no same-budget comparison with a standard linear compensator (SLA-style), RoFormer-style RoPE linear attention, all frequency pairs, or no rotation.
- RoPeSLR beats full attention on SC and MS at 80-90% sparsity (Tables 2 and 4), which suggests that fine-tuning on curated data shifts the metrics. Yet there is no fine-tuned full-attention reference.
- Only 5 of the 16 VBench dimensions are reported (no Dynamic Degree, no semantic dimensions), with no seeds or variance, and SVG2 runs at 72% / 80% sparsity rather than 90%.
- Latency against SLA is essentially the same (30.4 vs 31.2 s; 598.3 vs 601.0 s), so the paper stands or falls on this quality comparison.
Questions for authors
- Can you report the following variants with the same sparse branch, data and training budget: a standard linear compensator, RoFormer-style RoPE linear attention, all frequency pairs, and no rotation?
- Can you add a full-attention model fine-tuned on the same data for the same number of steps? Did SLA and VSA receive the same data and steps?
- How do SLA and RoPeSLR compare at 95% and 97% sparsity, given the "extreme sparsity" motivation?
- Which frequency pairs are in I^h, and how many? Small m corresponds to the highest rotational frequencies; why are these suited to a smooth global background?
- What are the block size and threshold, and how is the 90% ratio enforced? The low-rank branch sums over all tokens, including those already covered by the sparse branch; how is this double counting handled?