acceptodds
Under review as a conference paper at ICLR 2027

WRAP: Wigner Rotation-Equivariant Attention for Physical Signals on The Sphere

Abstract

Many physical signals are spherical: a radar signature is a function of viewing direction, and an atmospheric state is a field on a rotating planet. Projecting them onto a plane distorts area, distance and orientation and discards the rotational symmetry of the underlying physical system. Previous equivariant attention methods either rely on computationally expensive steerable kernels and tensor products, limiting scalability, or, are restricted to Euclidean space, planar images or camera poses rather than spherical data. We introduce Wigner-RoPE, a spherical rotary positional encoding that generalizes rotary positional embeddings from translations along a line to rotations on the sphere. Wigner-RoPE represents lifted spherical positions as rotations from a reference direction using Wigner D-matrices. Building on Wigner-RoPE, we develop Wigner rotation-equivariant attention for physical signals (WRAP), an expressive, multi-frequency, - equivariant attention mechanism compatible with efficient fused attention kernels. Stacking WRAP layers yields WRAP-former, a transformer for spherical physical signals. Across radar classification, spherical segmentation, and medium-range weather forecasting, WRAP-former outperforms strong baselines with alternative positional embeddings and symmetry–efficiency tradeoffs. Finally, we compare spherical embedding methods with varying degrees of symmetry breaking, finding that different symmetry choices benefit different physical domains.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Transformer, RoPE, Equivariance, Spherical, Radar, segmentation, weatherbench

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rAihOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Wigner-RoPE, a method that generalizes RoPE to spherical settings. Based on Wigner-RoPE, the authors develop WRAP-former, a Transformer for spherical data that is equivariant to specific rotation groups. WRAP-former is evaluated on radar classification, panoramic spherical segmentation, and medium-range weather forecasting. The results show that WRAP-former outperforms several strong baselines. In the weather forecasting task, the authors also discuss the impact of symmetry breaking on model performance.

Critical strengths
  1. This paper addresses an important question with clear motivation. By leveraging the properties of Wigner-D matrices, the authors elegantly extend RoPE to spherical settings, supported by sufficient theoretical proofs.
  2. The medium-range weather forecasting experiments are comprehensive. In addition to standard RMSE and ACC metrics, the authors analyze power at different spatial scales, as well as hydrostatic balance residuals and continuity equation residuals. They further discuss the impact of incorporating spherical symmetry on model performance.
Critical weaknesses
  1. The experiments on radar classification and panoramic spherical segmentation are somewhat insufficient, with relatively few baselines (e.g., HEAL-SWIN also uses the HEALPix grid and should be included as a baseline). The paper lacks evaluation results on rotated inputs, including rotations from specific rotation groups and arbitrary rotations (e.g., the rotated-input experiments on Spherical MNIST in Spherical CNNs).
  2. The main text does not seem to provide explicit experimental comparisons of the runtime of WRAP-former and the baselines, although compatibility with efficient fused attention kernels is mentioned as an advantage in the abstract.
Questions for authors
  1. As mentioned in W1, could the authors provide experimental results with more baselines, such as HEAL-SWIN and ViT with mixed-RoPE? Although the latter is a planar model, based on my experience, it also performs well on spherical tasks and is one of the commonly used approaches.
  2. As mentioned in W1, could the authors provide experimental results on rotated inputs?
  3. As mentioned in W2, could the authors provide a runtime comparison between WRAP-former and the baselines, or between WRAP and standard attention?
  4. Furthermore, the structure of RoPE ensures that the computational complexity of positional encoding is linear in both the number of query/key tokens and the embedding dimension. Does the use of Wigner-D matrices compromise the latter property? This may affect the efficiency of Wigner-RoPE.

Since there is no option between weak accept and clear accept this time, I can only give a weak accept for now. However, my position is between the two, and I may adjust my score based on the authors' response.

3 comments
AuthorsOpenReview ↗

Response to Reviewer rAih

Thank you for the positive assessment and the concrete suggestions, including HEAL-SWIN (Carlsson et al., 2024), which we have now run. Below, the SC lift and the frame lift are WRAP-former with one token per patch and with four.

W1 and Q1: "HEAL-SWIN also uses the HEALPix grid and should be included as a baseline" and "ViT with mixed-RoPE"

Added to Table 2: all four WRAP-former models outperform HEAL-SWIN. We trained it with the same recipe as every other row.

modelmIoUAcc.
HEAL-SWIN0.62796.48
S2 Transformer / S2 SegFormer (Bonev et al., 2025b)0.623 / 0.64596.5 / 96.8
WRAP-former (SC lift / frame lift), flat0.701 / 0.69297.30 / 97.20
WRAP-former (SC lift / frame lift), hierarchical0.681 / 0.68497.08 / 97.13

For translation/planar based RoPE, our main weather baseline is the axial form of Heo et al. (2024); we did not run the mixed form. The frame lift beats axial RoPE by 3.1% at 5-day z500 (306.7 against 316.6), and in every variant we ran (Table 6). Beyond these, Table 8 compares our weather model with the published WeatherBench 2 systems.

W1 and Q2: "The paper lacks evaluation results on rotated inputs"

Yes, we have added both kinds (Section 5.1 and App. J). On radar, under the grid's quarter turns, both WRAP-former models keep their accuracy exactly, while ViT (Dosovitskiy et al., 2021), Attn. on Sphere (Bonev et al., 2025b) and SFNO (Bonev et al., 2023) lose accuracy. Under the larger group D4D_4, the frame lift stays the most accurate (0.8296). For arbitrary rotations, on weather, the frame lift's forecast changes least under 300 random rotations (median 0.20, against 0.34 for axial RoPE). Figure 15 of the revision shows the error at every rotation.

W2 and Q3: "could the authors provide a runtime comparison between WRAP-former and the baselines, or between WRAP and standard attention?"

Yes, both. Against standard attention, FlashAttention (Dao et al., 2022) runs unchanged, with 2.5–7% extra time per layer, and the SC lift trains as fast as axial RoPE (0.680 against 0.688 s per step). Against published models on the same GPU, the SC lift is the fastest per step: 21.9 ms, against 73.8 ms for Stormer (Nguyen et al., 2024), which has the same width, depth and heads. Our 10-day forecast takes 1.06 s, against 5.00 s for Stormer and 6.64 s for SFNO on its 0.25∘0.25^\circ grid. Figure 13 of the revision plots error against inference compute.

Q4: "Does the use of Wigner-D matrices compromise the latter property?"

No. The head dimension is fixed at 64, and widening the model adds heads, not larger matrices. So the cost per head is fixed (64×6464 \times 64 multiply-adds), and the total grows linearly in the embedding dimension: at width 1024 it is 6% of the projection that produces each query and key. Each token is rotated independently, as in RoPE, so the cost is linear in the number of tokens too.

You said your assessment lies between weak and clear accept. We hope the new baselines and measurements answer your questions and support moving to the higher score, and we would be glad to provide more.

References

  • Bonev et al. (2023). Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere. ICML.
  • Bonev et al. (2025b). Attention on the Sphere. NeurIPS.
  • Carlsson et al. (2024). HEAL-SWIN: A Vision Transformer On The Sphere. CVPR.
  • Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
  • Heo et al. (2024). Rotary Position Embedding for Vision Transformer. ECCV.
  • Nguyen et al. (2024). Scaling Transformer Neural Networks for Skillful and Reliable Medium-Range Weather Forecasting. NeurIPS..

Thank you for the detailed response. I find the additional experimental results valuable, particularly the evaluations under rotated inputs and the runtime comparisons, which provide further evidence of WRAP's advantages.

Regarding computational complexity, I would like to clarify that the claimed linear scaling with the embedding dimension relies on keeping the attention head dimension fixed. This is somewhat different from standard RoPE, whose positional encoding cost remains linear in the head dimension regardless of its size. In practice, the head dimension is not necessarily fixed when scaling up models (e.g., from ViT-L to ViT-H). Therefore, I believe this assumption should be made explicit when discussing the computational complexity of Wigner-RoPE.

Overall, the authors have addressed several of my concerns with additional evidence, and I maintain my positive assessment of the paper. As mentioned in my original review, my assessment lies between Weak Accept and Clear Accept. Since there is no intermediate rating option, I will maintain my original score of 3, although my position remains more positive than a typical Weak Accept.

Thank you for the careful follow-up, and for clarifying the point on complexity of Wigner-RoPE relative to standard RoPE.

"the claimed linear scaling with the embedding dimension relies on keeping the attention head dimension fixed"

You are right: the cost is linear in the width only at a fixed head dimension. App. C.2 of the revision states this, and we have added there how the cost grows when the head dimension dhd_h grows. RoPE rotates pairs of channels, so it costs O(dh)O(d_h) per head for any dhd_h. Wigner-RoPE's blocks have size 2ℓ+12\ell+1 for degrees ℓ\ell up to about dh\sqrt{d_h}, so per head it costs about 43dh3/2\tfrac{4}{3}d_h^{3/2} multiply-adds in block form and dh2d_h^2 in the dense form we run. Over the D/dhD/d_h heads of a token of width DD, that is about 43Ddh\tfrac{4}{3}D\sqrt{d_h} or DdhD d_h, linear in DD only at a fixed dhd_h, as in all our models (dh=64d_h = 64).

One thing to note here is that the overhead stays bounded relative to the layer, though. The projection that forms each head's query or key costs DdhD d_h per token, so the dense rotation costs dh/D=1/Hd_h/D = 1/H of it for HH heads, whatever the head dimension. In your example, ViT-L and ViT-H both have 16 heads (Dosovitskiy et al., 2021), so the rotation stays at 6.25% of the projection as dhd_h grows from 64 to 80. At dh=64d_h = 64 we measured 2.5–7% extra time per attention layer (App. I).

We are glad the rotated-input and runtime results were useful, and we thank you for the constructive discussion and engagement with us in improving the paper as well as your your positive assessment.

References

  • Dosovitskiy et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR.
Reviewer_TF9fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WRAP-former for physical signals on the sphere. It introduces Wigner-RoPE to encode the relative positions and orientations of spherical patches. It introduces SC-lift and frame lift. Experiments on radar classification, spherical segmentation, and medium-range weather forecasting show competitive performance against the baselines.

Critical strengths
  1. Wigner-RoPE allows attention to consider both feature embeddings and the relative positions and orientations of spherical patches.

  2. The experiments cover radar classification, spherical segmentation, and medium-range weather forecasting. WRAP-former achieves competitive performance across these tasks, supporting its applicability to different spherical physical signals.

Critical weaknesses
  1. The frame lift uses more tokens than the SC-lift, so it is not clear how much of the gain comes from exact equivariance.

  2. The full models only guarantee exact equivariance to C4 or D4. The paper states that they are approximately equivariant under general SO(3) rotations, but does not quantify the equivariance errors.

Questions for authors

In Table 2, the frame lift performs worse than the SC-lift with the flat transformer and only slightly better with the hierarchical SegFormer. Could authors discuss what might explain these results and when they would recommend the frame lift over the SC-lift?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TF9f

Thank you for your review and clear questions. Below, the SC lift and the frame lift are WRAP-former with one token per patch and with four.

W1: "The frame lift uses more tokens than the SC-lift, so it is not clear how much of the gain comes from exact equivariance."

The gain does not come from the extra compute or from learned symmetry against augmentation baselines, but we cannot split it between the tokens and exactness.

  • At 24k steps the frame lift matches the SC lift's full 288k run with 20% fewer FLOPs.
  • Doubling the one-token models' width (3.3×3.3\times the FLOPs per step) does not help.
  • Axial RoPE (Heo et al., 2024) trained with D4D_4 augmentation scores 320.8 at 5-day z500, against 306.7.

The four tokens are the same patch in four orientations and they are how the frame lift becomes exact on D4D_4 while being compatible with standard linear layers, nonlinearities, and normalization used in transformers. So the extra tokens and the exact symmetry come together, but our experiments rule out compute and learned symmetry, not one of the two. A new lift that doesn't have this additional token cost would have different properties - namely weight sharing would have to be there, so it would be a different lift with different properties altogether.

W2: "The paper states that they are approximately equivariant under general SO(3) rotations, but does not quantify the equivariance errors."

We now quantify them, in the new App. J. WRAP itself is exactly SO(3){SO}(3)-equivariant (Prop. 2), and WRAP-former's error comes from sampling the sphere on a grid and lifting it to rotations, as in other models on discretized spheres. The SC lift is exact on C4C_4 and the frame lift on D4D_4, while the baselines' forecasts change by 5–40% under the same rotations. Under 300 random rotations, the frame lift's forecast changes least: a median of 0.20, against 0.34 for axial RoPE. Figure 15 shows where on the rotation group each model is and is not equivariant.

Q1: "Could authors discuss what might explain these results and when they would recommend the frame lift over the SC-lift?"

On 2D3DS, the two lifts are within 0.01 mIoU of each other, so the frame lift is not clearly worse. We think the reason is that the panoramas are canonicalized: every test image has the same orientation as the training images, with the floor below. On such data, models with less equivariance can match equivariant ones (Lawrence et al., 2026), and for this reason we deliberately build the frame lift's half-turned views by interpolation here, so it is exact only under quarter turns. Rotation shows the difference. On radar, the two lifts are close upright, but under D4D_4 the frame lift keeps 0.8296 while the SC lift drops to 0.7064. Weather is canonically oriented too, but the half turn is a real symmetry of the forecast (App. H.1), and there the frame lift is 2.7% more accurate.

We would recommend the frame lift when inputs can arrive in other orientations, or when the task has that symmetry, as weather does, and the SC lift for canonically oriented data, or when inference cost matters most.

We hope this answers your questions. If it does, we would be grateful if you would consider raising your score, and we would be glad to discuss any points further. References

  • Heo et al. (2024). Rotary Position Embedding for Vision Transformer. ECCV.
  • Lawrence et al. (2026). To Augment or Not to Augment? Diagnosing Distributional Symmetry Breaking. ICLR.
AuthorsOpenReview ↗

Follow up with Reviewer TF9f

Thank you again for your detailed review and feedback, which we feel has improved the paper. Here is a brief summary of how we addressed your questions in our response.

  • Extra tokens or exact equivariance (W1). The frame lift's gain is not from compute or from learned symmetry. After 24k steps the frame lift matches the SC lift's full 288k-step run with 20% fewer FLOPs, and a one-token baseline trained with D4D_4 augmentation stays behind (320.8 against 306.7 at 5-day z500). The four tokens are what make D4D_4 exact, so the two come together by construction.
  • Equivariance error (W2). App. J now measures it under 317 rotations. Each lift is exact on its group, and the frame lift has the lowest error under random rotations.
  • The two lifts in Table 2 (Q1). They are within 0.01 mIoU, which we attribute to the canonically oriented panoramas. Under rotated inputs the frame lift is more robust: under D4D_4 it keeps 0.8296 radar accuracy, where the SC lift drops to 0.7064. We recommend the frame lift when inputs can arrive in other orientations or the task has the symmetry, and the SC lift otherwise.

Could you let us know whether we addressed your concerns and answered your questions? If anything remains unclear, we would be glad to discuss it before the discussion period ends.

Thank you for the detailed response and additional experiments. The authors have addressed my concerns, and I would therefore like to raise my score to 3.

Reviewer_FG9LOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on modelling spherical data (e.g., weather). It is motivated by existing methods either treating spherical data as flat grids, leading to distortions and requiring the model to learn rotational symmetries from the data, or encoding symmetry into the architecture in a way that leads to computational overheads. The paper proposes relative spherical positional encodings (extending RoPE to spherical rotations) and an associated spherical attention mechanism that is compatible with existing transformer architectures and optimized attention kernels, making the method efficient. The relative position encoding requires a lifting operation and the paper uses two variants, a spherical coordinate lift (one token per patch) and a frame lift (four tokens per patch). The two lifts differ in what symmetries they are equivariant to and how expensive they are. The paper evaluates the approach across three different spherical problems (radar classification, spherical segmentation and weather forecasting).

Critical strengths
  • There is a real need for efficient, principled spherical modelling approaches. The proposed method is carefully motivated, provides partial SO(3) equivariance guarantees and runs efficiently on existing attention kernels.
  • The proposed method is evaluated across a broad range of spherical tasks and performs competitively with established methods, although errors are large and improvements are not always statistically meaningful.
Critical weaknesses
  • A large part of the motivation for the paper is that it is efficient. The paper should report model sizes and training/inference costs across all compared methods. This then lends itself to a discussion of the associated accuracy-efficiency trade-offs.
  • The paper should discuss more clearly the benefits of choosing their approach over equally well performing alternatives. For example, on radar classification, performance of WRAP-former with SC-lift essentially ties the ViT baseline, both outperforming the spherical baselines. The best performing model on the task is WRAP-former with frame lift but this comes with additional compute costs. Similarly, on the weather forecasting task the method is competitive with published models but does not outperform all of them. In the controlled comparison, WRAP-former with SC-lift is on par with axial RoPE and the performance benefit again comes from using the more expensive frame lift.
  • The equivariant claims are overstated. The models are not fully SO(3) rotation equivariant, and this should be made much more clear.
  • It's not clear whether it's the proposed encoding rather than the extra compute of the lifting that is responsible for the gains.
  • Referencing of the background literature on AI models on the sphere is not extensive, and numerous papers are missed.
Questions for authors
  • What do the error bars in Tables 1 and 2 represent?
  • What is the equivariance error of the model considered? These have been reported in many other papers on equivariant models on the sphere, but not this one.
4 comments
AuthorsOpenReview ↗

Response to Reviewer FG9L (1/2)

Thank you for a detailed review, and for pressing us on cost, equivariance error and related work, which all led to new results. Below, the SC lift and the frame lift are WRAP-former with one token per patch and with four.

W1: "The paper should report model sizes and training/inference costs across all compared methods."

Added, in App. I. The SC lift costs the same as the baselines: 0.680 s per training step, against 0.688 for axial RoPE (Heo et al., 2024), and it is the fastest model we timed at inference. The frame lift costs more per step (5.4 times in training) but needs fewer steps: at 24k steps it matches the SC lift's full 288k run with 20% fewer FLOPs. Its 10-day forecast takes 1.06 s, a fifth of Stormer's (Nguyen et al., 2024). Figures 11 and 13 of the revision plot error against training and inference compute for every model.

W2: "The paper should discuss more clearly the benefits of choosing their approach over equally well performing alternatives."

Added to Sections 5 and 6. There are three benefits:

  • Radar. Under quarter turns both WRAP-former models keep their accuracy exactly. The parameter-matched ViT, which is better upright (0.871), falls to 0.649 under quarter turns and to 0.562 under D4D_4.
  • Weather. Exact symmetry beats learned symmetry. With the same group as augmentation, axial RoPE scores 319.8 (C4C_4) and 320.8 (D4D_4), against 315.3 for the SC lift and 306.7 for the frame lift.
  • Published systems. Among models at about our 1.5∘1.5^\circ resolution, ours is the most accurate deterministic forecast in Table 8 from day 5 onwards: z500 at 5 days is 284, against 286 for Stormer and 289 for ArchesWeather (Couairon et al., 2024). The models ahead of ours run on finer 0.25∘0.25^\circ or 0.7∘0.7^\circ grids or, for WeatherMesh-4 (WindBorne Systems, 2025), train on years after 2020.
AuthorsOpenReview ↗

Response to Reviewer FG9L (2/2)

W3 and Q2: "The equivariant claims are overstated." and "What is the equivariance error of the model considered?"

You are right that WRAP-former is only approximately SO(3){SO}(3)-equivariant, and we have made this explicit in the abstract. WRAP, the attention mechanism, is exactly SO(3){SO}(3)-equivariant (Prop. 2). WRAP-former samples the sphere on a grid and lifts it to rotations, which introduces an equivariance error, as in other equivariant models on discretized spheres such as spherical CNNs and attention on the sphere (Cohen et al., 2018; Bonev et al., 2025b). On HEALPix, it stays exact on the rotations that permute the pixels: C4C_4 for the SC lift and D4D_4 for the frame lift on weather (Section 4.1, Theorem 1, Prop. 3). The new App. J measures the error under all other rotations. Each lift is exact on its own group, while the baselines' forecasts change by 5–40% under the same rotations. Under 300 random rotations, the frame lift has the lowest error (median 0.20, against 0.34–0.62). Figure 15 shows the error at all 317 rotations at a glance.

W4: "It's not clear whether it's the proposed encoding rather than the extra compute of the lifting that is responsible for the gains."

It is not the compute. At 24k steps the frame lift matches the SC lift's full 288k run with 20% fewer FLOPs, and giving the one-token models more compute does not help:

  • Twice the width (3.3×3.3\times the FLOPs) gives 317.5, against 315.3 at width 1024. These controls were already in Table 6 of the submission.
  • 4.8 times the steps (the full 288k schedule) buys the SC lift only 1.0%.

Figure 11 of the revision shows this directly. The four tokens add no parameters and no input: they are how the frame lift becomes exact on D4D_4.

W5: "Referencing of the background literature on AI models on the sphere is not extensive, and numerous papers are missed."

We agree, and have roughly doubled the references, widening the related work from rotary encodings to the full range of models on the sphere (App. K). This changes how we position the paper, not what it claims. Equivariant spherical models usually either pay for exact rotation symmetry with specialized operations, or drop it to keep standard architectures. WRAP sits between the two: it keeps a standard transformer and fused attention, is exact on the symmetries the grid allows, and we measure what it gives up elsewhere. The revised related work describes each family by where it sits on this trade-off.

Q1: "What do the error bars in Tables 1 and 2 represent?"

Standard deviations: over three training seeds in Table 1, and across test panoramas in Table 2. Both captions now say so. On your broader point about significance: on weather, the frame lift's lead over the SC lift is significant, since paired-bootstrap 95% intervals exclude zero at every lead from 1 to 10 days (5-day z500: −2.7% [−3.4, −2.0]). On radar, some upright differences are within the spread across seeds, but under rotation the gaps are large (Table R4).

We hope these results address your concerns. If they do, we would be grateful if you would consider raising your score, and we would be glad to discuss any point further.

References

  • Bonev et al. (2025b). Attention on the Sphere. NeurIPS.
  • Cohen et al. (2018). Spherical CNNs. ICLR.
  • Couairon et al. (2024). ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting. arXiv:2412.12971.
  • Heo et al. (2024). Rotary Position Embedding for Vision Transformer. ECCV.
  • Nguyen et al. (2024). Scaling Transformer Neural Networks for Skillful and Reliable Medium-Range Weather Forecasting. NeurIPS.
  • WindBorne Systems (2025). WeatherMesh-4 Technical Blog. https://windbornesystems.com/blog/weathermesh-4-technical-blog
AuthorsOpenReview ↗

Follow up with Reviewer FG9L

Thank you again for your detailed review and engagement, which we feel has directly improved the paper. Here is a brief summary of how we addressed your concerns in our response.

  • Cost (W1, W4). App. I now compares our models' size and cost with the baselines and published systems. The frame lift's gain is not its extra compute: after 24k steps it matches the SC lift's full 288k-step run with 20% fewer FLOPs.
  • Benefits over alternatives (W2). Under quarter turns both WRAP-former models keep their radar accuracy exactly, while the best upright baseline falls from 0.871 to 0.649.
  • Equivariance (W3, Q2). The abstract now says WRAP-former is approximately SO(3)\mathrm{SO}(3)-equivariant, and App. J measures the error: each lift is exact on its group, and the frame lift has the lowest error under random rotations.
  • Related work and error bars (W5, Q1). App. K roughly doubles the references, and the table captions now define the error bars.

Could you let us know whether these address your concerns? If anything remains unclear, we would be more than happy to discuss it before the discussion period ends.

I thank the authors for their detailed response and revision. They’ve addressed my concerns about the overstated equivariance claims, expanded the literature and explained the errors bars in the tables. They also show that the frame lift’s gain is not explained by additional width or compute and the new experiments demonstrate the method is robust to rotations of the data. I will revise my score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3) - summary of the reviews and our responses

We thank the reviewers for careful and constructive reviews. Our paper introduces WRAP, which generalizes RoPE from encoding relative position as translations on the plane to rotations on the sphere for such physically structured signals: queries, keys and values are rotated by Wigner-D matrices of each token's orientation, so attention depends only on relative rotations, and WRAP is exactly SO(3)\mathrm{SO}(3)-equivariant (Prop. 2). WRAP-former, the transformer built from it, samples the sphere on the HEALPix grid (Górski et al., 2005) and lifts it to rotations, which, as in other equivariant models on discretized spheres, makes it approximately SO(3)\mathrm{SO}(3)-equivariant and exact on a group of pixel-permuting rotations that depends on the lift. The SC lift gives each patch one token and is exact under the grid's quarter turns (C4C_4). The frame lift gives each patch four tokens, the same patch in four orientations with shared weights, and on weather is exact under all of D4D_4, the rotations that map the grid onto itself. On radar and segmentation, whose data come in one canonical orientation, we deliberately build its half-turned views by interpolation instead, which keeps every view faithful and leaves the half turn approximate (panel 3). To make the discussion easy to follow, we summarize each review and our response below. Panels 2 and 3 give the evidence.

We encourage the reviewers to look at Figures 11, 13 and 15 of the revision, which show the main results at a glance: Figures 11 and 13 plot each model's error against its training and its inference compute, and Figure 15 maps each model's equivariance error over all 317 rotations.

Reviewer FG9L

  • Praise: "a real need for efficient, principled spherical modelling approaches"; the method is "carefully motivated, provides partial SO(3) equivariance guarantees and runs efficiently on existing attention kernels".
  • Concerns: cost reporting; whether the gains come from the encoding or from the frame lift's extra compute; overstated equivariance claims and unmeasured equivariance error; missing related work; error bars.
  • Response: We measured the cost of every model (App. I). The SC lift trains at the baselines' speed and is the fastest model we timed. Compute does not explain the frame lift's gain: at 24k steps it matches the SC lift's full 288k run with 20% fewer FLOPs, and wider or longer-trained one-token models do not catch up. We clarified in the abstract that WRAP is exactly SO(3)\mathrm{SO}(3)-equivariant and WRAP-former approximately so, measured WRAP-former's error under 317 rotations (App. J), roughly doubled the references, and defined the error bars.

Reviewer TF9f

  • Praise: Wigner-RoPE "allows attention to consider both feature embeddings and the relative positions and orientations of spherical patches", with competitive results "supporting its applicability to different spherical physical signals".
  • Concerns: whether the frame lift's gain comes from its extra tokens or from exact equivariance; the equivariance error is not quantified; the two lifts on 2D3DS (Armeni et al., 2017).
  • Response: The gain does not come from compute or from learned symmetry, and the tokens and the exact symmetry come together by construction. Each lift is exact on its group, and under random rotations the frame lift has the lowest error. On 2D3DS the two lifts are within 0.01 mIoU, which we explain, and we say when to use each.

Reviewer rAih

  • Praise: the authors "elegantly extend RoPE to spherical settings, supported by sufficient theoretical proofs", and the weather experiments are "comprehensive".
  • Concerns: more baselines, HEAL-SWIN (Carlsson et al., 2024) and mixed RoPE (Heo et al., 2024); rotated inputs; runtime; cost in the embedding dimension.
  • Response: All four WRAP-former models outperform HEAL-SWIN on 2D3DS. Under quarter turns, radar accuracy is unchanged where every baseline drops, and on weather the frame lift's error is the lowest. FlashAttention (Dao et al., 2022) runs unchanged, and the cost stays linear in the embedding dimension.

The revised paper now includes (changes marked in blue):

  • an abstract that states WRAP is exactly SO(3)\mathrm{SO}(3)-equivariant and WRAP-former approximately so, with the error measured;
  • the cost of every model, and figures of error against training and inference compute (App. I);
  • the equivariance error under 317 rotations (App. J);
  • related work roughly twice as extensive (App. K);
  • error bars defined in the captions of Tables 1 and 2, and HEAL-SWIN in Table 2;
  • the rotated radar results in Section 5.1 and App. J.1 (Table 17), and takeaways on when to use each lift in the conclusion.

We believe these results address the shared concerns, and we would be glad to discuss any of them further.

AuthorsOpenReview ↗

General response (2/3) - compute and cost

The new baselines are on WeatherBench 2, our largest and most expensive benchmark (a full training run of the SC lift takes 54 B200-hours), which is where the rebuttal's time and compute allowed us to run them. Weather scores are 5-day RMSE over all 732 initializations of 2020, after 48k training steps unless stated.

Does the frame lift's gain come from extra compute? No. It reaches the same accuracy with less compute, and extra compute does not help the one-token models.

  • Less compute, same accuracy. At 24k steps the frame lift matches the SC lift's full 288k schedule (z500 312.1 against 312.0) with 20% fewer training FLOPs.
  • Width. Doubling it (3.3×3.3\times the FLOPs per step) makes the one-token models slightly worse.
  • Steps. Training the SC lift 4.8 times longer, to the full 288k schedule, buys it only 1.0%. The frame lift leads by 2.5–2.8% at every length.
  • Augmentation. Axial RoPE (Heo et al., 2024) trained with D4D_4 augmentation scores 320.8, against 306.7.

What we cannot separate is the frame lift's tokens from its exact symmetry: the four tokens add no parameters and no input, and they are what makes D4D_4 exact (Prop. 3). Figure 11 of the revision shows this at a glance: at every training compute, the frame lift's error curve lies below the SC lift's.

Table R1. 5-day z500 RMSE (m$^2$ s$^{-2}$) at 48k steps. At 288k steps: frame lift 304.3, SC lift 312.0.

modelFLOPs / stepz500
frame lift7.8×7.8\times306.7
SC lift / width 20481×1\times / 3.3×3.3\times315.3 / 317.5
axial RoPE / width 20481×1\times / 3.3×3.3\times316.6 / 317.0
axial RoPE + C4C_4 / D4D_4 augmentation1×1\times319.8 / 320.8

What does WRAP cost? Little.

  • Attention. Queries and keys are rotated before the kernel, so FlashAttention (Dao et al., 2022) runs unchanged, at 2.5–7% extra time per attention layer.
  • Training. The SC lift runs at axial RoPE's speed (0.680 against 0.688 s per step). The frame lift takes 5.4 times as long per step, which the result above shows it repays.
  • Inference. The SC lift is the fastest model we timed. Our Table 8 model forecasts 10 days in 1.06 s, a fifth of Stormer's time (Nguyen et al., 2024), and matches or beats Stormer on z500 and T850 at every lead. Figure 13 of the revision shows the same at inference: width buys nothing, and only the frame lift lowers the error.

Table R2. Inference time on one GPU, each model with its released code. The published models are Stormer (Nguyen et al., 2024), MOSAIC (Zhdanov et al., 2026) and SFNO (Bonev et al., 2023).

model (grid)ms per steps per 10-day forecast
SC lift, ours (1.5∘1.5^\circ)21.90.20
frame lift, ours, Table 8 (1.5∘1.5^\circ)107.61.06
Stormer (1.4∘1.4^\circ)73.85.00
MOSAIC (1.5∘1.5^\circ)115.41.15
SFNO (0.25∘0.25^\circ)166.76.64
AuthorsOpenReview ↗

General response (3/3) - equivariance error and rotated inputs

Each lift is exactly equivariant on its grid group, and under all other rotations the frame lift has the lowest error.

WRAP is exactly SO(3){SO}(3)-equivariant (Prop. 2). WRAP-former is approximately so, because sampling the sphere on a grid and lifting it to rotations introduce an error, as in other models on discretized spheres. On the HEALPix grid, a model can be exact only under rotations that permute the pixels: C4C_4 for the SC lift (Theorem 1) and D4D_4 for the frame lift on weather (Prop. 3). We measured the error under every other rotation by rotating the input, taking one step and rotating the forecast back (new App. J).

  • On their groups, both lifts are exact to float32 precision, while the baselines' forecasts change by 5–40%.
  • Under 300 random rotations, the frame lift's forecast changes least (median 0.20, against 0.34–0.62).

Figure 15 of the revision maps each model's error at every one of the 317 rotations, and the frame lift's errors are the smallest overall. Table R3 summarizes it.

Table R3. Relative change of the weather forecast under rotation. The baselines are a learned absolute position embedding and axial RoPE (Heo et al., 2024).

modelquarter turnshalf turns300 random rotations (median)
learned embedding0.11–0.140.400.36
axial RoPE0.050.350.34
SC lift5×10−7\mathbf{5\times10^{-7}}0.580.62
frame lift1.3×10−6\mathbf{1.3\times10^{-6}}1.3×10−6\mathbf{1.3\times10^{-6}}0.20

Rotated radar inputs. We rotated the test inputs of the trained radar models. Under quarter turns both WRAP-former models keep their accuracy exactly, while every baseline loses accuracy, including the parameter-matched ViT that is best upright. On radar and segmentation the frame lift is exact on C4C_4 and approximately equivariant under the half turn, by design: for data in one canonical orientation, where the half turn is not a symmetry of the data, we build its half-turned views by interpolation so that every view stays a faithful rotation of the patch (Section 4.5 of the revision). Even so, under D4D_4 it loses the least accuracy of any model.

Table R4 (Table 17 of the revision). Radar classification accuracy under rotated test inputs. The baselines are ViT, Attn. on Sphere (Bonev et al., 2025b) and SFNO (Bonev et al., 2023).

modeluprightC4C_4D4D_4
WRAP-former (SC lift)0.84040.84040.7064
WRAP-former (frame lift)0.85290.85290.8296
ViT (param-matched)0.8710.6490.562
Attn. on Sphere0.84930.71050.5517
SFNO0.80590.76270.7376

References

  • Dao et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
  • Bonev et al. (2023). Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere. ICML.
  • Nguyen et al. (2024). Scaling Transformer Neural Networks for Skillful and Reliable Medium-Range Weather Forecasting. NeurIPS.
  • Zhdanov et al. (2026). (Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Models. ICML.
  • Carlsson et al. (2024). HEAL-SWIN: A Vision Transformer On The Sphere. CVPR.
  • Bonev et al. (2025b). Attention on the Sphere. NeurIPS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.