- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a new position embedding approach, S2RoPE, for transformers operating on the sphere by adapting Rotary Position Embeddings (RoPE) to this non-Euclidean space, leveraging spherical harmonics and therefore better preserving spherical distances. Two variants of S2RoPE corresponding to the two interpretations of RoPE are proposed: S2RoPE-SH, which multiplies query and key features by spherical harmonic values, and S2RoPE-Wigner, which rotates groups of features according to each token’s position. The authors then evaluate these two variants against three baselines (No RoPE, 2D RoPE, and GeoRoPE) in two setups using ERA5 data with different resolutions, geometries, and architectures, focusing on global weather forecasting.
Critical strengths
- The problem addressed is particularly important, as many Earth systems applications require good representations that preserve distances on the sphere.
- The approaches look sound and are well motivated by the background. Having two extensions following the two readings of RoPE is meaningful.
- Good performance on the two proposed setups.
Critical weaknesses
I think the scope of the paper and the evaluation of the approach are a bit limited:
- Many Earth systems applications beyond global weather forecasting, such as ocean circulation, atmospheric transport, or tropical cyclone tracking, could benefit from approaches like S2RoPE. I understand that, as an application-driven paper, the authors wanted to focus on a single task, but I also believe the paper would be stronger if it demonstrated S2RoPE's potential to generalize to one or two additional applications.
- In addition, even in the realm of weather forecasting, I think the experimental setups and baselines tested are slightly insufficient. Only two setups are considered, both using ERA5 data, and only three baselines are evaluated. In particular, a few mentioned related works (e.g., SpheRoPE, StereoRoPE) are not considered at all, despite being alternatives to S2RoPE.
- I also feel that the paper could be improved by providing additional experiments on where S2RoPE is stronger than alternatives. For example, explicitly considering Earth's curvature is known to be particularly beneficial at high absolute latitude. I would therefore be interested to see how the baselines and S2RoPE performed across different latitude ranges. An in-depth analysis of the input resolution and its impact on the different position embedding approaches could also be relevant.
Questions for authors
- Could you provide intuition and a corresponding experiment explaining why one variant could be preferred over the other?
- Is there a specific reason why you only considered the task of weather forecasting?
- Why are some baselines not considered for comparison?
I am willing to increase my rating if you address these questions and weaknesses well.
4 comments
4. Performance across latitude ranges and spatial token counts
We have added a latitude-resolved comparison against axial 2D RoPE in WeatherGenerator. We use this reference to examine how the gains over a latitude–longitude rotary encoding vary with latitude under matched training settings. All compared models retain global positional embeddings and use zero latent training noise.
We evaluate nine variables in 10° latitude bands using RMSE and a zonal spectral diagnostic over physical wavelengths of 500–5,000 km. The figure in the appendix subsection “Forecast performance by latitude” reports both metrics across all latitude bands and forecast leads.
Both S2RoPE variants improve high-latitude RMSE and zonal spectral error at longer leads. At day 14, over regions with absolute latitude at least 60°, S2RoPE-SH and S2RoPE-Wigner reduce RMSE relative to 2D RoPE by 3.13% and 2.66%, respectively, and zonal spectral error by 11.16% and 7.86%. Improvements also occur at lower latitudes, with patterns varying by variant and forecast lead. These results support benefits at high latitudes without implying that gains increase monotonically with latitude.
Our backbones use different spatial token counts—12,288 in WeatherGenerator, 2,048 in Stormer, and 450 in LaDCast per forecast state. These experiments do not isolate input-resolution effects, since architecture and training settings also differ.
5. When to prefer S2RoPE-SH or S2RoPE-Wigner
The two variants retain different aspects of classical RoPE. S2RoPE-SH applies harmonic multipliers independently to complex feature channels, changing both amplitude and phase. Its normalization controls average squared gain across channels but does not preserve arbitrary feature norms. S2RoPE-Wigner jointly transforms channels within complete degree blocks using orthogonal matrices, preserving query and key norms.
The experiments show different strengths rather than a universal ranking. In WeatherGenerator, SH achieves lower log-spectral error in the forecast windows beyond day 3, while Wigner achieves lower RMSE than SH at the reported leads from day 7 onward. In Stormer, Wigner achieves the lowest RMSE at every reported lead. At day 14, it retains an RMSE improvement over No RoPE, whereas SH performs worse than No RoPE in both RMSE and ACC. In LaDCast, Wigner outperforms SH in ensemble-mean RMSE, ACC, and CRPS at every reported lead. Cyclone tracking likewise shows that their relative ranking depends on lead time and the track estimator.
Between the two S2RoPE variants, Wigner is a reasonable starting choice when prioritizing RMSE, and it also performs best on CRPS in LaDCast. SH offers stronger spectral fidelity in WeatherGenerator, so the preferred variant depends on the backbone and evaluation objective. These are empirical preferences in the evaluated settings; norm preservation alone does not establish that Wigner will always perform better.
I would like to thank the authors for addressing my concerns by adding a new baseline, a task, and an additional experiment on performance by latitude. I will therefore increase my rating from 2 to 3.
Thank you for reconsidering the paper. Your suggestions on the additional baseline, the cyclone-tracking evaluation, and the latitude-resolved analysis have made the paper stronger. We appreciate your time and careful review.
Thank you for your review and for the suggestions on broadening the evaluation. We have expanded the evaluation with a third forecasting backbone, a StereoRoPE baseline, tropical cyclone tracking, and a latitude-resolved analysis. We also clarify the different strengths of the two S2RoPE variants.
1. Application scope and tropical cyclone tracking
We focused on global weather forecasting because it provides a demanding setting for evaluating positional representations on the sphere: forecasts must capture interactions across spatial scales and remain accurate during extended rollouts. The positional operators themselves use spherical token coordinates and do not depend on weather-specific variables, making them applicable in principle to other attention-based models on the sphere.
Following your suggestion, we have added tropical cyclone tracking using LaDCast ensemble forecasts. This evaluates a downstream application of the forecasts rather than a separately trained cyclone model. We use 20-member ensembles over the 2018 evaluation set, comprising 102 storms and 402 storm–initialization cases before lead-dependent eligibility filtering.
We evaluate two track estimators: tracking the ensemble-mean field, following the original LaDCast protocol, and tracking individual members before averaging their positions on the sphere. All methods use the same tracker configuration and eligible cases at each lead, with equal weighting of individual storms.
Both S2RoPE variants reduce track error relative to GeoRoPE at every reported lead under both estimators. At day 5, S2RoPE-Wigner reduces error by 20.66% under the original estimator and 15.96% under member-wise tracking and position averaging. The latter estimator also improves GeoRoPE itself, so the S2RoPE gains persist against this improved reference. StereoRoPE performs best at day 1 under both estimators, while an S2RoPE variant performs best at every reported lead from day 2 onward.
These results are included in the new appendix subsection “Tropical cyclone tracking” and summarized in the main text.
2. Additional forecasting architecture
We have added LaDCast, a probabilistic latent diffusion model, alongside WeatherGenerator and Stormer. Unlike the regression-based forecasting settings of the original experiments, LaDCast generates ensemble forecasts through iterative denoising in an autoencoder’s latent space. Training and evaluation settings are held fixed across positional encoding variants. We evaluate ensemble-mean RMSE and ACC, as well as CRPS.
The new LaDCast results include the requested StereoRoPE comparison. Each entry below gives the percentage changes in RMSE / CRPS relative to native GeoRoPE; negative values indicate lower error.
Both S2RoPE variants improve ensemble-mean RMSE, ACC, and CRPS relative to native GeoRoPE at all reported lead times. S2RoPE-Wigner achieves the best results among the evaluated methods across all three metrics and all reported leads (new Table 3). This broadens the architectural and forecasting settings of our evaluation.
3. Additional baselines
We have added a StereoRoPE adaptation in both LaDCast and WeatherGenerator. The LaDCast results are reported above; the WeatherGenerator comparison below uses the same zero-noise training settings as the other RoPE variants. Implementation details are provided in the appendix subsection “StereoRoPE implementation.” RMSE changes relative to No RoPE (DN, default noise), in %, are shown below.
Both S2RoPE variants achieve lower RMSE than StereoRoPE at all reported leads through day 10. At day 14, StereoRoPE lies between S2RoPE-Wigner and S2RoPE-SH. An S2RoPE variant has the highest ACC in every column (revised Table 1).
SpheRoPE was introduced as a training-free, inference-time adaptation of pretrained diffusion transformers for panoramic generation. Our experiments instead train forecasting variants with the selected positional encoding. We prioritized StereoRoPE because it was introduced for atmospheric forecasting; SpheRoPE remains discussed in the related work but is not included in the current experimental comparison.