acceptodds
Withdrawn from ICLR 2027

Seasonality-Aware Hybrid Convolutional Transformer for Antarctic Sea Ice Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Antarctic sea ice concentration (SIC) forecasting is an important yet challenging task due to the coexistence of complex spatial structure, long-range temporal dependencies, and strong seasonal variability. Conventional convolution-based models are effective at capturing local spatial patterns, but often have limited ability to model long-term temporal evolution. To address these challenges, we build on a hybrid convolutional transformer forecasting framework for monthly Antarctic SIC forecasting. This framework combines convolutional encoding for spatial feature extraction with space-time factorised self-attention for SIC modelling. We further introduce two season-based mechanisms: a month-aware positional encoding that injects calendar-month information into the token representation, and a seasonal temporal bias that encourages attention to periodically related historical states. Experimental results show that the proposed framework achieves better performance than convolutional and recurrent baselines as well as ECMWF's physics-based dynamical model SEAS5 under both classification and regression metrics. Ablation studies further indicate that the seasonality-aware components provide consistent additional gains in both short- and long-horizon prediction. These results demonstrate the value of combining convolutional structures, attention mechanisms, and periodic prior information for Antarctic SIC forecasting.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Transformer, Convolutional neural network, Sea ice prediction, Spatio-temporal attention, Autoregressive models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_m6N8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a seasonality-aware hybrid convolutional Transformer framework for monthly Antarctic sea ice concentration (SIC) forecasting. The framework combines UNet-based spatial feature extraction with factorized spatio-temporal self-attention to capture local spatial patterns and long-range temporal dependencies. To explicitly incorporate seasonal information, the authors introduce two components: Month-aware Positional Encoding (MonthPE), which uses Fourier-based embeddings to represent annual periodicity, and Seasonal Temporal Attention Bias (SeasonBias), which incorporates a learnable periodic bias into temporal attention scores. The model uses 12 months of historical SIC observations to autoregressively forecast up to 12 future months. Experiments on the NSIDC Antarctic SIC dataset compare the proposed approach against statistical, convolutional, recurrent, and dynamical forecasting baselines, using both regression and classification metrics. The reported results demonstrate competitive forecasting performance, while ablation studies and attention visualizations provide additional evidence for the effectiveness of the proposed seasonality-aware components.

Critical strengths
  1. The proposed MonthPE and SeasonBias mechanisms explicitly incorporate seasonal periodicity into feature representations and temporal attention, respectively. These designs are well motivated by the pronounced annual variability of Antarctic sea ice and provide an interpretable way to introduce domain-specific prior knowledge into the forecasting framework.

  2. The hybrid architecture combines convolutional encoding and decoding with factorized spatio-temporal self-attention at a compressed feature resolution. This design enables the model to capture both local spatial structures and long-range temporal dependencies while maintaining computational feasibility for high-resolution SIC forecasting.

  3. The authors evaluate the proposed framework against statistical, dynamical, convolutional, and recurrent forecasting baselines using multiple regression and classification metrics. The experiments cover forecasting horizons of up to 12 months, while ablation studies across two backbone architectures and attention visualizations provide additional evidence supporting the effectiveness of the proposed seasonal components.

Critical weaknesses
  1. The proposed framework primarily combines existing techniques, including CNN/UNet architectures, factorized spatio-temporal attention, Fourier-based positional encoding, and cosine-based attention bias. Although these components are reasonably integrated for Antarctic SIC forecasting, the methodological innovations appear incremental. The paper does not sufficiently distinguish MonthPE and SeasonBias from existing periodic encoding and relative positional bias mechanisms.

  2. The effectiveness of the proposed seasonal priors is not fully substantiated. In particular, SeasonBias is formulated solely based on the relative temporal distance between input months, and the default 12-month input window does not explicitly enable comparisons between the same calendar months across different years.

  3. The paper lacks comparisons with alternative seasonal encoding strategies or conventional relative positional biases, leaving it unclear whether the improvements arise from explicitly modeling seasonal periodicity or simply introducing additional temporal inductive biases.

  4. The experimental evaluation mainly focuses on conventional convolutional and recurrent baselines, with limited comparisons against more recent spatio-temporal forecasting architectures.

  5. The paper does not provide detailed computational complexity analysis, such as parameter counts and FLOPs.

  6. The ablation results reveal inconsistent improvements: for example, MonthPE alone degrades the one-month prediction performance of the UNet-based model, while combining MonthPE and SeasonBias does not consistently outperform MonthPE alone at the 12-month horizon.

Questions for authors

Please responds to my concerns in the weaknesses.

Reviewer_4TJmOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a model for Antarctic sea ice concentration (SIC) forecasting that combines a convolutional encoder-decoder with factorized spatio-temporal attention. It incorporates seasonal information through MonthPE, a Fourier-based calendar-month positional encoding, and SeasonBias, a periodic bias added to the temporal attention logits.

The model uses the previous 12 monthly SIC fields to predict the next month and repeats this process autoregressively to forecast the next 12 months. The authors evaluate the separate and combined effects of the two seasonal components on CNN and U-Net backbones. They also compare the models with neural and statistical baselines, as well as SEAS5 over its available forecast horizon. In Table 2, the combined setting improves all reported metrics over the corresponding baseline without the seasonal components at both lead months 1 and 12 on both backbones.

Critical strengths
  1. The motivation for incorporating calendar-month information and the annual cycle into Antarctic sea ice forecasting is clear. MonthPE and SeasonBias introduce this information into the token representations and temporal attention logits, respectively.

  2. Applying attention at a compressed bottleneck and separating temporal and spatial attention is a reasonable design for reducing the computational cost of high-resolution inputs while modelling spatio-temporal dependencies.

  3. The authors evaluate the separate and combined effects of the seasonal components on two backbones. The improvements over the baseline across all reported metrics at lead months 1 and 12 support the usefulness of the combined design.

  4. The comparisons include statistical baselines and SEAS5 in addition to neural models, allowing the proposed models to be assessed against different forecasting approaches.

Critical weaknesses

1. Architectural differences from earlier factorised spatio-temporal attention models

The paper explicitly states that it follows the divided space-time attention design of TimeSformer [1]. In spatio-temporal forecasting, STAEformer [2] also applies temporal attention followed by spatial attention and uses calendar-based periodicity embeddings, while STTN [3] applies a spatial module followed by a temporal module. Applying temporal and spatial attention sequentially is therefore not a new architectural contribution by itself.

The convolutional bottleneck, gridded sea ice data, and autoregressive forecasting setup differ from those in these studies. Even with these differences, the paper does not clearly separate the shared design choices from the new components. This makes it difficult to assess what is new in the overall architecture.

2. Relationship between MonthPE and existing time representations

Periodic time embeddings have been used in earlier work, including Time2Vec [4]. GMAN [5] and STAEformer [2] also use calendar-based time representations based on the day of the week and time of day. Calendar-time and thermal-time positional encodings have also been compared for satellite image time series [6].

MonthPE is not mathematically identical to these methods. In particular, Time2Vec learns frequencies and phase shifts and includes a non-periodic linear term, whereas MonthPE combines fixed annual Fourier harmonics through a learnable linear projection. However, the paper does not sufficiently establish the additional benefits of this choice over existing time representations. A controlled comparison with a simple learned month embedding, for example, would help separate the benefit of providing seasonal information from the benefit of using the Fourier representation.

3. Relationship between SeasonBias and existing periodic relative attention biases

PENGUIN [7] introduces a relative attention bias based on known periods. The idea of directly incorporating periodic information into query-key scores therefore needs to be discussed in relation to this earlier design.

There are important differences between the two methods. SeasonBias uses a cosine bias for a single annual cycle with a learnable coefficient. PENGUIN uses a piecewise-linear bias based on circular distance, predefined head-specific slopes, and a grouped structure for multiple periods. The methods should therefore not be treated as identical. However, the paper does not clearly explain the additional benefits of these differences or of choosing a cosine bias. The methodological contribution of SeasonBias relative to the earlier design remains unclear.

References

[1] Bertasius, G., Wang, H., & Torresani, L. (2021). Is Space-Time Attention All You Need for Video Understanding? Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR, 139, 813-824. https://proceedings.mlr.press/v139/bertasius21a.html

[2] Liu, H., Dong, Z., Jiang, R., Deng, J., Deng, J., Chen, Q., & Song, X. (2023). Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting. Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM), 4125-4129. https://doi.org/10.1145/3583780.3615160

[3] Xu, M., Dai, W., Liu, C., Gao, X., Lin, W., Qi, G.-J., & Xiong, H. (2020). Spatial-Temporal Transformer Networks for Traffic Flow Forecasting. arXiv preprint, arXiv:2001.02908. https://arxiv.org/abs/2001.02908

[4] Kazemi, S. M., Goel, R., Eghbali, S., Ramanan, J., Sahota, J., Thakur, S., Wu, S., Smyth, C., Poupart, P., & Brubaker, M. (2019). Time2Vec: Learning a Vector Representation of Time. arXiv preprint, arXiv:1907.05321. https://arxiv.org/abs/1907.05321

[5] Zheng, C., Fan, X., Wang, C., & Qi, J. (2020). GMAN: A Graph Multi-Attention Network for Traffic Prediction. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 34(1), 1234-1241. https://doi.org/10.1609/aaai.v34i01.5477

[6] Nyborg, J., Pelletier, C., & Assent, I. (2022). Generalized Classification of Satellite Image Time Series With Thermal Positional Encoding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1391-1401. https://doi.org/10.1109/CVPRW56347.2022.00145

[7] Sun, T., Chen, Y., & Sun, W. (2026). PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting. Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR, 300, 703-711. https://proceedings.mlr.press/v300/sun26b.html

Questions for authors

Clarification 1: Differences from earlier designs

Could the authors clarify the new contributions relative to STAEformer [2], Time2Vec [4], GMAN [5], and PENGUIN [7]? In particular, what additional benefits do the fixed annual Fourier representation and the learnable cosine attention bias provide over the related designs?

Clarification 2: Training runs

How many independent training seeds were used for each setting? Do the reported results come from a single training run or an average over multiple runs?

Clarification 3: Test sample construction

Which forecast start dates make up the 48 test samples in Table 7? Are the same test samples used in Tables 1 and 2 and Figure 2, and does the set of samples change with the forecast lead time?

Clarification 4: Additional benefits of seasonal attention

Do the attention pattern changes in Figure 3 also help interpret temporal dependencies or improve robustness under distribution shifts? If these are claimed benefits of the method, could the authors provide supporting analyses or experimental results?

Additional comparison 1: Choice of time representation and attention bias

Are there comparisons under the same backbone and training conditions that replace MonthPE with a simple learned month embedding or replace SeasonBias with another periodic relative attention bias? These comparisons would help separate the effect of providing seasonal information from the effect of the particular representation and bias, and would directly inform the assessment of the methodological contribution.

Additional comparison 2: Two-step loss

Are there results comparing a one-step prediction loss with the current two-step prediction loss under the same model and training conditions? If so, how does this choice affect autoregressive forecasting performance over lead months 1-12?

Suggested additional experiment 3: Shorter input window

When each model is trained using four months of historical observations rather than 12 months, does the model with both MonthPE and SeasonBias still improve performance over the same backbone without these components across lead months 1-12? If results using the same forecast start dates and test samples are available, could the authors provide them? This question is intended to assess whether the benefits of the seasonal components remain when the input does not cover a full annual cycle.

Reviewer_M6BCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors use a CNN/UNet to encode monthly sea ice fields, incorporating low-resolution factorized spatiotemporal attention, MonthPE, and cosine-based SeasonBias. The model recursively forecasts 12 months from the preceding 12 months. Comprehensive experiments verified the effectiveness of the key elements, MonthPE and SeasonBias. Detailed comments are as follows:

Critical strengths
  1. The paper tries to address an important task: sea ice concentration prediction, and the experiments demonstrate the effectiveness of the proposed model.
  2. The authors provide the detailed parameters of the proposed method to ensure reproducibility.
Critical weaknesses
  1. My main concern is the novelty. The authors apply the UNet for sea ice concentration prediction. Based on this, the authors propose MonthPE and SeasonBias; the previous one is a position encoding, and the latter is actually a simple relative time bias. The seasonal bias in Eq. 3 depends only on relative month offsets and does not encode absolute calendar information, which is provided by MonthPE. Moreover, a 12-month input contains no cross-year same-month pairs, so the attention patterns cannot establish same-month retrieval across years. Comparisons with relative positional encodings, recency bias, and learned month embeddings are needed to validate its contribution.

  2. SeasonBias does not consistently improve performance. For UNet, adding SeasonBias to MonthPE worsens the 12-month RMSE from 0.1026 to 0.1033 and slightly reduces macro F1. Similarly, the CNN model shows a higher 12-month RMSE with SeasonBias.

  3. For the spatiotemporal prediction model, the authors should try some more recent models beyond PredRNN, such as TAU and other recent models. This may also influence the final conclusion of this work.

Overall, I think this work is okay for an application paper; the methodological novelty is really a concern. Based the applciation value and methodalicacal novelty of this work, I don't think this submission meet the stahdard of the ICLR.

Minor issue: Please unify the symbols and indices of the equations.

Questions for authors
  1. Please provide the efficiency metrics such as FLOPs, memory demand, and inference time.
  2. Please reply to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.