acceptodds
Under review as a conference paper at ICLR 2027

Autoregressive Embedding Prediction for Temporal Earth Observation Foundation Models

Abstract

Multi-spectral remote sensing time series contain rich temporal and spectral signals. However, existing remote sensing foundation models often struggle to learn robust temporal semantics from these data due to their sensitivity to observation-specific noise and insufficient modeling of fine-grained temporal structure. We propose AREO, a feature-predictive foundation model that addresses these limitations by learning semantic representations while preserving fine-grained temporal structure throughout the prediction process. We introduce Autoregressive Embedding Prediction, in which a causal predictor, combined with temporally consistent high-ratio masking, estimates each masked patch's representation from current and preceding context. This design avoids noisy observation-level reconstruction and instead encourages temporally stable, spatially localized semantic representations. Pretrained on SSL4EO-S12, AREO is evaluated under both frozen-backbone and full fine-tuning protocols across three temporal and two static datasets. It outperforms the baselines on the temporal benchmarks under both evaluation protocols and achieves competitive results on static tasks. These findings suggest that temporally directed embedding prediction can support transferable representations for Earth observation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Remote Sensing, Foundation Model, Earth Observation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_avUEOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The article proposes a method for training a foundation transformer model for remote sensing data, a popular task in the community right now. It is heavily inspired by general self-supervised methods, such as MAE, JEPA, and DINO. These methods provide two versions of the data to a network: one with full context and the other with limited context. They force the network to infer the full context (or its representation) from the limited context.

The article proposes applying these principles to remote sensing time series, specifically Sentinel 2 time series. The article's originality comes from two points. First, the final guess is made at the frame level, not at a collapsed representation. Second, the guess is performed in an autoregressive manner, which is like adding further masking (i.e., masking the future). Otherwise, the architecture is the classic one-student with autograd and limited context, as well as a frozen EMA teacher with full context. On top of the student, a predictor attempts to infer the full context representation from the limited context version. This part of the architecture is causal, using causal masking in the attention matrices. Until now, this is similar to JEPA; however, the loss is similar to DINO rather than MSE or L1. Representations are projected onto prototypes with two heads and a softmax so that cross-entropy loss can be applied. As in DINO, centering is applied to the teacher's scores for stability.

The method is evaluated against a large set of other methods on different downstream tasks in two settings: frozen + trained head and fully fine-tuned. In both settings, the proposed method outperforms the others. The authors also provide an extensive ablation study to justify their choices.

Critical strengths

The article is easy to follow and understand for the most part, except for a few points that I will mention in the section on weaknesses. It uses widely known general tools and applies them to remote sensing, which is probably why it is easy to follow.

The main idea I find interesting is showing that while applying these methods to remote sensing, causal prediction (predicting the future) brings large improvements compared to unconstrained prediction. This is quite interesting because, in video MAE or video JEPA, they find that it degrades the foundation model's ability to do so. This is not illogical, as the structure and change of remote sensing time series are quite different from classical ones. In fact, I think the authors didn't emphasize this point enough and should have.

The evaluation and ablation study are very complete and they demonstrate the superiority of the method in comparison to competing methods.

Critical weaknesses

I think it's a fair criticism to say that the method section of this article is heavily inspired by well-known methods. It's mostly the experimental results that give the article significance. Even though this should not be the only criterion for rejection, the lack of real novelty is a major weakness.

There are also some things that are unclear, which I will ask about in the questions.

Questions for authors

There are important details that I can't find in the article :

  1. Does the encoder network process each frame independently, and is time like a batch dimension? Or, is there an inter-frame attention mechanism like in video backbones? If there is inter-frame attention, then the prediction might not be completely causal because the early tokens could contain information about future tokens from the encoding. In that case, the main claim about causality would be greatly weakened.

  2. For me, the attention matrix comes after the softmax. Therefore, the mask should be additive and have a value of 1 for current and past tokens and a value of 0 for future tokens. It seems that the mask is added before the softmax to the query-key product, like a bias. I think the authors should clarify how this is done.

  3. Figure 4 might be more interesting if you extended the masking above 90% to show that you have reached optimal points. As it is, it could still grow.

  4. In the method section, Objective, I gues s that the prototyping and centering of the teacher are comming from DINO, if so it should be cited

Reviewer_64onOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The motivation of this work is practically relevant. Multispectral remote sensing time series contain rich information about land surface changes, but are also affected by cloud contamination and variations in imaging conditions. AREO retains per-timestamp, per-patch feature representations within a masked feature prediction framework and introduces a predictor with a temporal attention mask to encourage the student model's predicted features to match the feature distribution of the EMA teacher. The paper reports consistent performance improvements across three temporal downstream tasks, and the overall presentation is coherent. However, significant limitations remain in terms of technical novelty, methodological completeness, and the rigor of the experimental evidence. First, components such as EMA teachers, prototype distribution alignment, cross-temporal masking, and latent feature prediction have all been explored in prior work. The main contribution of this paper lies in combining these components and retaining per-timestamp feature representations throughout the prediction stage. However, the current experiments do not sufficiently demonstrate that this combination establishes a clearly novel temporal learning mechanism. Second, key technical details remain unspecified, and the implementation of causal prediction raises a serious concern about potential information leakage, which may undermine the soundness of the method and the reliability of the experimental conclusions. In addition, several numerical and design inconsistencies exist between the baseline results and the ablation experiments, weakening the reproducibility and persuasiveness of the reported findings. More importantly, the current ablations do not adequately disentangle the individual contributions of causal prediction, the prototype learning objective, and the masking strategy. The paper also lacks direct validation of its claimed temporal semantic modeling capability and noise robustness mechanism. Therefore, the current results primarily support the practical value of AREO as an effective training configuration, but are insufficient to establish its methodological novelty or provide convincing evidence for the temporal modeling mechanism claimed by its core design.

Critical strengths
  1. Addresses the bidirectional shortcut problem in remote sensing time series: static backgrounds can encourage bidirectional predictors to exploit interpolation shortcuts, and the paper provides compelling evidence through convergence curves (Fig.5) and performance comparisons (Table 3).
  2. Provides strong evidence for the effectiveness of high-ratio temporally consistent masking: Fig.4 and Table2 demonstrate substantial performance gains from 90% temporally consistent masking, with a 3.8 percentage point improvement in mIoU under the frozen-backbone setting on PASTIS. These results highlight the importance of preventing cross-frame spatial information leakage for latent prediction in remote sensing time series.
  3. Adopts a rigorous evaluation protocol: By reporting results under both full fine-tuning and frozen-backbone settings across multiple PANGAEA benchmarks, the paper effectively assesses the transferability of the learned self-supervised representations.
Critical weaknesses
  1. The scope of the causal constraint is unclear. The paper applies a causal mask only to the predictor but does not specify whether the encoder permits bidirectional attention across timestamps. If it does, the representation of the current frame may already contain information from future frames, in which case masking the predictor alone cannot guarantee temporal causality throughout the entire prediction pathway. If it does not, the paper should explain how the backbone models cross-frame relationships. The authors should clarify the information flow within the encoder and validate the causal claim by perturbing future frames.
  2. The ablation studies in Fig.4 and Table2 primarily serve to identify an effective training configuration and provide limited evidence of methodological novelty. Prior studies have already investigated the effects of high masking ratios and the choice between temporally consistent masking and independent masking.
  3. Table.3 lacks a control experiment that combines the causal predictor with a continuous feature regression loss. Consequently, although the existing experiments can assess the effect of the causal predictor on downstream performance while keeping the prototype-based objective fixed, they cannot independently quantify the contributions of the predictor design and the learning objective, or their interaction. The reported mIoU results are also insufficient to attribute robustness to cloud noise to either design choice.
  4. Appendix A.1 and Table.5 indicate that DynamicEarthNet provides only four-band RGB and near-infrared imagery from PlanetFusion, whereas AREO and the baseline methods use different spectral bands during pretraining. However, the paper does not explain how the near-infrared channel is mapped to B8A in AREO, nor does it specify the actual channels retained and the zero-padding applied for each model. Differences in band mapping and the proportion of zero-padded channels may introduce varying degrees of distribution shift, thereby affecting the interpretability of the cross-model comparisons in Table.1. The authors should provide the band mapping for each model and conduct additional sensitivity experiments using a unified set of input bands.
  5. The claims of robustness to clouds, haze, and observation noise lack direct validation. The abstract and introduction identify resistance to clouds, haze, and atmospheric disturbances as key motivations, yet the experiments do not report performance stratified by cloud coverage or provide robustness curves under controlled cloud occlusion or radiometric perturbations. The mIoU scores for the four groups of examples in Fig.3 reflect segmentation performance on only a limited number of samples and cannot distinguish robustness gains from general improvements in segmentation performance. The authors should compare performance degradation under varying perturbation levels on the same samples and evaluate the results against the main baselines.
  6. The temporal transfer protocol is insufficiently described. AREO is pretrained on four-frame sequences, but the paper does not specify the number of frames used for downstream evaluation on PASTIS and DynamicEarthNet, the frame sampling strategy, or how date encoding is applied to inputs of different lengths. If the six-frame configuration adopted by PANGAEA is used, the current results only evaluate transfer from four-frame pretraining to six-frame inputs and do not support claims about modeling longer temporal sequences. The authors should clarify the frame selection rules for each dataset, specify whether the pretrained causal predictor is used during downstream inference, and report performance with varying numbers of input frames to distinguish the benefits of temporal modeling from the effects of the sampling strategy.
Questions for authors

In the rebuttal, the authors should address the specific requests raised at the end of Core Concerns 1, 3, 4, 5, and 6 by providing relevant clarifications and supplementary experiments. Specifically, they should clarify the cross-frame attention mechanism in the backbone encoder, report ablation results for combining the causal predictor with continuous feature regression loss, detail the band mapping and zero-padding schemes used by each model, provide quantitative stratified evaluations of robustness to clouds and haze, and clarify the number of frames sampled for downstream evaluation while correcting the numerical errors in Table.1.

2 comments

Thank you for the detailed response. The new experiments and clarifications address many of my concerns, but I remain unconvinced by the causal interpretation. As the authors explain, the encoder attends bidirectionally across time; only the predictor is causally masked. Consistent masking prevents direct copying of the target patch from another frame, but tokens encoded at or before time t may still carry information from later frames. The predictor’s attention mask therefore does not establish that its prediction at t is independent of future observations. I change the score to 2.

AuthorsOpenReview ↗

Thank you for your response

Dear Reviewer 64on,

We sincerely thank you for raising the score. We believe that your feedback on contamination levels, band inputs, and longer input sequences has helped us further clarify our motivation and strengthen the experimental evidence.

Regarding your remaining concern, AREO operates at the patch level, and consistent 90% masking hides each target patch location at every timestamp. Therefore, future information reaches the predictor only indirectly, through the very sparse visible patches. This context helps with individual observations that are hard to interpret in isolation due to noise or ambiguity. Therefore, our model can capture fine-grained temporal semantics while remaining robust to observation-specific noise. Our experiments support this design: encoding frames independently reduces frozen-backbone performance by about 3.8 to 7.0 mIoU (Table 3).

Thank you once again for your time in reviewing our paper and rebuttal.

Best regards,

Authors

Reviewer_koPBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AREO, a self-supervised feature-predictive foundation model for multispectral Earth observation time series. It keeps per-frame patch tokens and, under temporally consistent high-ratio masking, trains a causal predictor to match the prototype distributions of an EMA teacher at masked locations. Pretrained on SSL4EO-S12 and evaluated on PANGAEA, it outperforms existing foundation models on three temporal datasets and is competitive on two static ones.

Critical strengths
  • Unlike prior JEPA-style EO models that collapse the temporal sequence before prediction, AREO preserves patch-level temporal structure throughout pretraining, which directly targets a clear limitation of existing work.

  • AREO improves over the strongest baseline on all temporal benchmarks under both frozen-backbone and fine-tuning protocols, and the paper includes ablations on masking ratio, masking method, temporal prediction strategy and predictor configuration.

Critical weaknesses
  • The central claim that causal embedding prediction drives the gains rests on Table 3, but the compared variants appear undertuned. CEP, which pools features over time as AnySat does, reaches only 21.9 mIoU on PASTIS compared with 39.2 for AnySat in Table 1, and UEP even falls below the MAE baseline. Against such weak references, the 11.6 point gap between UEP and AEP cannot reliably establish the benefit of the proposed design.

  • Limited novelty and unclear motivation of the causal design The framework largely combines existing components, namely iBOT/DINO-style masked patch self-distillation with an EMA teacher and tube masking, with a temporal causal mask in the predictor as the main new element. Moreover, the encoders appear to attend over the full sequence, so causality is enforced only in a predictor discarded after pretraining, and consistent masking already removes the shortcut used to motivate the causal design.

  • Robustness to observation-specific noise such as clouds is a central motivation of the paper, yet it is supported only qualitatively by Figure 3, without any quantitative evaluation under varying levels of contamination.

  • The conclusion states that AREO achieves the best performance on all five benchmarks, whereas TerraFM and Satlas outperform it on m-bigearthnet under fine-tuning. The term autoregressive is misleading, as the predictor recovers masked patches of the current frame rather than forecasting future observations. In addition, all ablations are reported from a single seed.

Questions for authors

Please refer to weakness.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period is coming to an end, I kindly ask all reviewers who have not yet responded to the authors' rebuttal to carefully review their responses and provide feedback or update their reviews accordingly.

Thank you for your time and efforts!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.