acceptodds
Under review as a conference paper at ICLR 2027

EarthMind: Persistence-Anchored Latent Transitions for Robust Multimodal Earth Observation Forecasting

Abstract

Forecasting Earth observations is hampered by a practical fact: the inputs are routinely incomplete. In our globally distributed benchmark, only 90.3% of Sentinel-2 months and 82.6% of Sentinel-1 months are usable. We present EarthMind, a multimodal spatiotemporal world model that frames monthly forecasting as learning a transition from a persistence-anchored state rather than regenerating the scene from scratch. EarthMind fuses Sentinel-2 multispectral imagery, Sentinel-1 SAR, static terrain, and ten weather variables into latent Earth states. It predicts the next state with a Transformer and decodes that state into future Sentinel-2 and Sentinel-1 imagery and weather. Two extensions target real-world use. Structured modality masking (EarthMind-Robust) reproduces realistic sensor outages during training. Band-wise calibration toward persistence (EarthMind+) learns how much of the predicted change each spectral band can be trusted with. On 1,000 sites with site-disjoint splits (16,562 test sequences), EarthMind, trained from scratch, performs comparably to the pretrained Prithvi-EO-2.0 foundation model on clean inputs, with no significant difference on any metric, while additionally forecasting SAR and weather. EarthMind+ attains the lowest RMSE (0.0676), highest PSNR (23.40 dB) and SSIM (0.9704), and lowest SAM (4.68) among the evaluated methods, while Prithvi retains the lowest MAE. When SAR and weather are both missing, structured masking reduces RMSE by 35.3%. In four-month closed-loop rollouts, EarthMind stays 42.8% below last-observation-carried-forward, and its frozen latent state improves land-cover balanced accuracy by 12.2 points over raw features without any land-cover supervision. The learned calibration coefficients further identify the shortwave-infrared bands as the least predictable.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
world models, multimodal learning, missing modality robustness, Earth observation, remote sensing, spatiotemporal forecasting, self-supervised representation learning, sensor fusion

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_dp2nOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

EarthMind forecasts monthly optical imagery, SAR and weather from incomplete multimodal histories using persistence-anchored latent transitions. Structured modality masking improves robustness to sensor outages; ensembling and calibration improve optical forecasting. Evaluation uses site-disjoint splits across 1,000 locations.

Critical strengths

The setting addresses realistic sensor gaps and tests forecasting at unseen locations. Structured masking reduces RMSE by 35.3% under joint SAR and weather loss, at a 2.3% increase on clean inputs, relative to standard EarthMind (Section 5.3).

Critical weaknesses

W1. The qualitative evaluation is inadequate. The main paper provides no visual comparisons of predictions and ground truth. The supplementary material also contains very limited examples, making it difficult to assess the model's forecasting quality. In addition, supplementary examples, including Tiles 442 and 508, contain regular color bands and grid textures. These may be visualization artifacts, but it is unclear whether the underlying training or evaluation tensors are affected. Section 4 also permits patches with only 5% valid pixels, without explaining invalid-pixel handling for the reported metrics, particularly SSIM. The figures and masking protocol need clarification before Table 1 can be interpreted confidently.

W2. The central mechanism lacks direct validation. Figure 5 removes input modalities and the Transformer, but does not isolate fused-state anchoring, the explicit latent difference or latent-consistency supervision. Consequently, the reported ablations provide limited support for the paper's central design claim. Matched direct-prediction and persistence-residual controls are needed to establish whether the proposed mechanism contributes beyond these simpler alternatives.

Questions for authors

Q1. Can you provide corrected optical examples with valid-pixel masks for the same samples, clarify whether training/evaluation data are affected, and specify masking for each metric?

Q2. Can matched ablations separate the effects of anchoring and latent-consistency supervision from direct prediction or a simpler persistence-residual model under identical input, training and evaluation settings?

Q3. How does EarthMind perform on scenes with substantial temporal changes? Can the authors provide evidence that the model learns meaningful temporal dynamics rather than simply relying on persistence?

Reviewer_toY3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The article is about forecasting remote sensing time series. More specifically, it is about robust forecasting and multimodal forecasting (S1, S2, and weather). The authors explain that not all data in most datasets is usable. This means that a model should be robust even when there is a month of missing data. They explain that, to build a robust model, it should predict the variation from previous states rather than predicting it from scratch.

The authors propose a model architecture that uses one acquisition per modality for the last four months to predict the next one. The model replaces unavailable samples with a trainable placeholder. Once all modalities are combined, the method has a global latent representation vector for each month, which is processed by a transformer. The last vector is used to predict the next month's latent vector, which is then used to predict the increment to add to the current month's non-averaged representation. This representation is then decoded to create the future S1, S2, and weather.

The authors propose two alternatives: the robust one, which is trained with a probability of masking some available samples, and another one. The second alternative is an ensemble model built from three models trained with different seeds and combined bandwise with weights fitted on the validation set.

The authors compare the reconstruction of their method with that of others and also provide an ablation study.

Critical strengths

The idea of treating monthly EO forecasting as a prediction of a transition based on the current state rather than a regeneration of the entire scene is interesting. Handling missing modalities realistically is also a relevant practical concern.

Critical weaknesses
  1. The gain is marginal and the comparison is unfair. As the authors’ own statistical test shows, Single EarthMind is not significantly better than Prithvi-EO-2.0 or ConvLSTM. EarthMind+ produces the best results, adding a three-seed ensemble and per-band shrinkage toward persistence. These are both generic, posterior procedures that could be applied to any baseline, so the comparison is misleading.

  2. The core idea remains not tested. There is no variant without the anchored residual, so the benefit of anchoring is never isolated.

  3. The objective combines six terms with handmade set of weights, and none of them are ablated. This is particularly important for the latent-consistency terms, which support the world-model framing. Why use different losses, such as MSE and cosine distance, to supervise the same thing?

  4. The visual examples in the supplementary material are disturbing. We only see flat colors or demarcations, not a landscape. Most of the understandable examples don't show any changes that can be predicted. There is no explanatory text that could help.

Questions for authors

Could you report results with ensembling and calibration applied to the baselines, and an ablation without the anchored residual?

Reviewer_qXEdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the practical challenge that Earth Observation (EO) data are often multimodal, temporal, and incomplete. It proposes an availability-aware multimodal state representation, masking strategies, and a persistence-anchored latent transition mechanism to predict future states rather than regenerating scenes from scratch. However, it remains unclear how existing methods handle these challenges and what limitations remain. In the experiments, the proposed model (EarthMind) outperforms Prithvi-EO-2.0-100M-TL on several metrics.

Critical strengths
  • The motivation is clear and the concept of adapting the world model concept to the earth observation forecasting is interesting
  • The persistence-anchored latent transition is an intuitive approach to forecasting future observations by learning changes relative to the previous state rather than reconstructing the entire scene from scratch
Critical weaknesses
  • The paper is poorly written and the novelty is very limited. EarthMind addresses missing or incomplete observations in EO forecasting through separate encoders and masking methods. However, these methods are not new [2, 3]. There is no discussion of whether existing methods [2, 3, 4, 5] can handle this problem and what are remaining problems. Furthermore, the characterization of EarthMind as a world model is not well justified. A world model should learn state transitions conditioned on external information (e.g., weather) in the latent space [1], but I do not see this concept in the Figure 2. The persistence-anchored latent transition seems to be the only contribution of the paper. However, its novelty relative to existing residual prediction or change-based forecasting approaches remains unclear.

  • The experiment comparison is weak and unclear about the fairness. The experiments are conducted on the paper's constructed dataset, and it is unclear whether the performance gains over Prithvi come from the training dataset or the proposed model design. Furthermore, the fine-tuning settings for Prithvi are not clearly described. For example, Prithvi only receives multispectral inputs, whereas the proposed model uses multimodal inputs, making the comparison potentially unfair. In addition, the experiments miss several important baselines (e.g Contextformer [4], Earthformer [5], Skysense[3]) or Prithvi-EO-2.0-300M, etc.

Reference:

[1] Mastering diverse control tasks through world models, Nature 2025

[2] RobSense: A Robust Multi-modal Foundation Model for Remote Sensing with Static, Temporal, and Incomplete Data Adaptability, CVPR 2025

[3] SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery, CVPR 2024

[4] Multi-modal Learning for Geospatial Vegetation Forecasting, CVPR 2024

[5] Earthformer: exploring space-time transformers for earth system forecasting, NeurIPS 2022

Questions for authors
  1. How does the proposed model perform on public benchmarks, such as EarthNet2021?
  2. How is Prithvi fine-tuned in the experiments? For example, what input modalities, multispectral bands, and fine-tuning parameters, etc are used?
  3. Why is only one-step Sentinel-2 forecasting used for comparison?
  4. How are the loss weights in Equation (3) chosen?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.