- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Wigner-RoPE, a method that generalizes RoPE to spherical settings. Based on Wigner-RoPE, the authors develop WRAP-former, a Transformer for spherical data that is equivariant to specific rotation groups. WRAP-former is evaluated on radar classification, panoramic spherical segmentation, and medium-range weather forecasting. The results show that WRAP-former outperforms several strong baselines. In the weather forecasting task, the authors also discuss the impact of symmetry breaking on model performance.
Critical strengths
- This paper addresses an important question with clear motivation. By leveraging the properties of Wigner-D matrices, the authors elegantly extend RoPE to spherical settings, supported by sufficient theoretical proofs.
- The medium-range weather forecasting experiments are comprehensive. In addition to standard RMSE and ACC metrics, the authors analyze power at different spatial scales, as well as hydrostatic balance residuals and continuity equation residuals. They further discuss the impact of incorporating spherical symmetry on model performance.
Critical weaknesses
- The experiments on radar classification and panoramic spherical segmentation are somewhat insufficient, with relatively few baselines (e.g., HEAL-SWIN also uses the HEALPix grid and should be included as a baseline). The paper lacks evaluation results on rotated inputs, including rotations from specific rotation groups and arbitrary rotations (e.g., the rotated-input experiments on Spherical MNIST in Spherical CNNs).
- The main text does not seem to provide explicit experimental comparisons of the runtime of WRAP-former and the baselines, although compatibility with efficient fused attention kernels is mentioned as an advantage in the abstract.
Questions for authors
- As mentioned in W1, could the authors provide experimental results with more baselines, such as HEAL-SWIN and ViT with mixed-RoPE? Although the latter is a planar model, based on my experience, it also performs well on spherical tasks and is one of the commonly used approaches.
- As mentioned in W1, could the authors provide experimental results on rotated inputs?
- As mentioned in W2, could the authors provide a runtime comparison between WRAP-former and the baselines, or between WRAP and standard attention?
- Furthermore, the structure of RoPE ensures that the computational complexity of positional encoding is linear in both the number of query/key tokens and the embedding dimension. Does the use of Wigner-D matrices compromise the latter property? This may affect the efficiency of Wigner-RoPE.
Since there is no option between weak accept and clear accept this time, I can only give a weak accept for now. However, my position is between the two, and I may adjust my score based on the authors' response.
3 comments
Thank you for the detailed response. I find the additional experimental results valuable, particularly the evaluations under rotated inputs and the runtime comparisons, which provide further evidence of WRAP's advantages.
Regarding computational complexity, I would like to clarify that the claimed linear scaling with the embedding dimension relies on keeping the attention head dimension fixed. This is somewhat different from standard RoPE, whose positional encoding cost remains linear in the head dimension regardless of its size. In practice, the head dimension is not necessarily fixed when scaling up models (e.g., from ViT-L to ViT-H). Therefore, I believe this assumption should be made explicit when discussing the computational complexity of Wigner-RoPE.
Overall, the authors have addressed several of my concerns with additional evidence, and I maintain my positive assessment of the paper. As mentioned in my original review, my assessment lies between Weak Accept and Clear Accept. Since there is no intermediate rating option, I will maintain my original score of 3, although my position remains more positive than a typical Weak Accept.
Thank you for the careful follow-up, and for clarifying the point on complexity of Wigner-RoPE relative to standard RoPE.
"the claimed linear scaling with the embedding dimension relies on keeping the attention head dimension fixed"
You are right: the cost is linear in the width only at a fixed head dimension. App. C.2 of the revision states this, and we have added there how the cost grows when the head dimension grows. RoPE rotates pairs of channels, so it costs per head for any . Wigner-RoPE's blocks have size for degrees up to about , so per head it costs about multiply-adds in block form and in the dense form we run. Over the heads of a token of width , that is about or , linear in only at a fixed , as in all our models ().
One thing to note here is that the overhead stays bounded relative to the layer, though. The projection that forms each head's query or key costs per token, so the dense rotation costs of it for heads, whatever the head dimension. In your example, ViT-L and ViT-H both have 16 heads (Dosovitskiy et al., 2021), so the rotation stays at 6.25% of the projection as grows from 64 to 80. At we measured 2.5–7% extra time per attention layer (App. I).
We are glad the rotated-input and runtime results were useful, and we thank you for the constructive discussion and engagement with us in improving the paper as well as your your positive assessment.
References
- Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR.
Response to Reviewer rAih
Thank you for the positive assessment and the concrete suggestions, including HEAL-SWIN (Carlsson et al., 2024), which we have now run. Below, the SC lift and the frame lift are WRAP-former with one token per patch and with four.
W1 and Q1: "HEAL-SWIN also uses the HEALPix grid and should be included as a baseline" and "ViT with mixed-RoPE"
Added to Table 2: all four WRAP-former models outperform HEAL-SWIN. We trained it with the same recipe as every other row.
For translation/planar based RoPE, our main weather baseline is the axial form of Heo et al. (2024); we did not run the mixed form. The frame lift beats axial RoPE by 3.1% at 5-day z500 (306.7 against 316.6), and in every variant we ran (Table 6). Beyond these, Table 8 compares our weather model with the published WeatherBench 2 systems.
W1 and Q2: "The paper lacks evaluation results on rotated inputs"
Yes, we have added both kinds (Section 5.1 and App. J). On radar, under the grid's quarter turns, both WRAP-former models keep their accuracy exactly, while ViT (Dosovitskiy et al., 2021), Attn. on Sphere (Bonev et al., 2025b) and SFNO (Bonev et al., 2023) lose accuracy. Under the larger group D4, the frame lift stays the most accurate (0.8296). For arbitrary rotations, on weather, the frame lift's forecast changes least under 300 random rotations (median 0.20, against 0.34 for axial RoPE). Figure 15 of the revision shows the error at every rotation.
W2 and Q3: "could the authors provide a runtime comparison between WRAP-former and the baselines, or between WRAP and standard attention?"
Yes, both. Against standard attention, FlashAttention (Dao et al., 2022) runs unchanged, with 2.5–7% extra time per layer, and the SC lift trains as fast as axial RoPE (0.680 against 0.688 s per step). Against published models on the same GPU, the SC lift is the fastest per step: 21.9 ms, against 73.8 ms for Stormer (Nguyen et al., 2024), which has the same width, depth and heads. Our 10-day forecast takes 1.06 s, against 5.00 s for Stormer and 6.64 s for SFNO on its 0.25∘ grid. Figure 13 of the revision plots error against inference compute.
Q4: "Does the use of Wigner-D matrices compromise the latter property?"
No. The head dimension is fixed at 64, and widening the model adds heads, not larger matrices. So the cost per head is fixed (64×64 multiply-adds), and the total grows linearly in the embedding dimension: at width 1024 it is 6% of the projection that produces each query and key. Each token is rotated independently, as in RoPE, so the cost is linear in the number of tokens too.
You said your assessment lies between weak and clear accept. We hope the new baselines and measurements answer your questions and support moving to the higher score, and we would be glad to provide more.
References