acceptodds
Under review as a conference paper at ICLR 2027

Colocated MTG - EarthCARE ML-ready dataset for deriving the 3D structure of the atmosphere

Abstract

Clouds are a critical component of the Earth’s climate system, acting as the main modulator of energy and radiation, with a major influence on climate and meteorology. Their complex 3D structure and dynamics pose serious observational and modelling challenges. The recently launched Earth Cloud, Aerosol and Radiation Explorer (EarthCARE) satellite mission carries passive and active instruments that give information on both the horizontal and vertical structure of the atmosphere and was specifically designed to provide improved observations and synergistic products that address these challenges. At the same time, the Meteosat Third Generation (MTG) Flexible Combined Imager (FCI) provides geostationary multispectral observations over Europe, Africa and the surrounding regions in high temporal resolution (every 10 minutes). In recent years, numerous studies have sought to bridge the gap between active sensors that capture detailed vertical cloud properties but sample sparsely in time (e.g., CALIPSO and CloudSat, with a 16-day repeat cycle, and now EarthCARE, with a 25-day repeat cycle) and geostationary imagers that observe clouds continuously yet resolve only their top (e.g., MSG, Himawari, GOES). This effort is driven by the growing need to incorporate 3D cloud information into meteorological forecasting, energy-yield estimation, radiative transfer modelling, and related applications. In this work, we introduce the first ML-ready dataset of spatially and temporally colocated MTG-EarthCARE scenes of clouds, designed to bridge these complementary observational capabilities and allow for further atmospheric and machine learning studies. The dataset covers the full 2025 period and includes approximately 1.9 million MTG pretraining scenes and 167,407 finetuning patches across three temporal colocation windows (2, 3, and 4 minutes), over a geographic extent limited to ±45° around the MTG subsatellite point to minimize parallax effects. Each sample consists of fixed-size 128×128 patches of MTG FCI multispectral bands along with the MTG FCI cloud mask and the EarthCARE vertical profile of cloud types and cloud microphysical properties corresponding to the overpass. The dataset is provided in three different versions, each accounting for a different maximum temporal colocation window, enabling the impact of temporal alignment on downstream ML tasks to be systematically investigated. The dataset also supports applications such as vertically resolved retrievals of cloud types and microphysical properties from passive geostationary sensor observations. To establish initial benchmarks, we provide ML baselines for all dataset versions on these tasks, evaluated using F1 score and IoU for cloud type classification and MSE, SSIM, and PSNR for microphysical property retrieval. Beyond these retrieval tasks, the dataset enables applications such as cloud representation learning and the investigation of the relationship between two-dimensional radiances and the underlying vertical cloud structure. Code and datasets will be made publicly available on GitHub.

Primary area:
datasets and benchmarks
Keywords:
machine learning, dataset, baseline, clouds, atmosphere

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HTSLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a dataset which focuses on understanding the 3D structure of clouds. Observations from two sensors are combined, one which provides dense data without explicit 3D structure and another which provides sparse 3D structure. The idea would be to use the dense data to predict the sparse samples and thus be able to generate a dense wall-to-wall map of estimated 3D structure. A baseline U-Net across a few settings and more limited results from a ViT model (with MAE pretraining) are presented.

Critical strengths
  • The paper is well written.
  • The goal of better understanding climate drivers is noble.
  • The selection of baseline methods is sensible.
  • The data split strategy seems sensible given the nature of the data.
Critical weaknesses

Summary:

This is going to be a short review, because I honestly don't think much of the content of the paper is really relevant to the ICLR community. So, I'm going to focus on how I think about the value of a dataset paper. First, it should be well explained and as open as possible. This paper does that. Second, it should ideally have some significant potential value as a signifier of progress in data interpretation (e.g., strong performance on this dataset should have some real-world impact, either directly on this problem or the indirectly on other problems). This is a little weaker, because we can't really see how performance improvements on the proposed metrics will impact "estimates of climate sensitivity". Third, it should motivate the development of novel methods in representation learning or provide an interesting evaluation case that will help us better understand existing methods. This is the weakest element of this work. The paper does not articulate this and I don't see any obvious ways in which this is likely to lead to novel methods in representation learning. I can see it being a good target for hyperparameter tuning of existing methods, but I don't think there will be impact outside the remote sensing communit.

Specific weaknesses:

  • This seems more like a remote sensing paper than an ICLR paper. It's is unclear whether this proposed benchmark would lead to any truly novel innovation in the representation learning domain that couldn't already be handled by other similar problems in remote sensing (e.g., canopy height estimation comes to mind as a problem that has a similar sparse-training dense-inference setup, but I'm sure there are many). This is evidenced by the citations, which are dominated by remote sensing papers with very little that indicates the recent work in representation learning that could be impacted by the release of this dataset.
  • The evaluation metrics are sensible from an ML perspective, but they don't seem to connect to the task in a meaningful way. A claimed motivation for this in the climate science domain is that it will help resolve some ambiguities in the inputs to current models, but nothing about the evaluation really tests the impact of this.
  • It's unclear how meaningful the improvements in metrics across different baseline settings are relative to just using the sparse labels and dense labels directly as inputs to a simulation model.
  • Stated differently: Are the PSNR/SSIM/F1 numbers good? Are they bad? How far are we from having estimates that would be useful?
  • One additional concern is that the experiments appear to be incomplete. For example, the unet is evaluated at three temporal offsets, but the ViT model is only evaluated at one.
Questions for authors

Are there other dataset papers that have been published at ICLR that you think are similar in nature to this paper? How would compare the potential and actual impact on the ICLR community of a few of these previous papers to this paper?

Reviewer_MnPZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarthCAREBench, a dataset that spatially and temporally aligns 2D multispectral imagery from the MTG satellite with vertical cloud structure observations from EarthCARE, providing cloud-type labels and cloud microphysical properties. The authors further demonstrate its potential for 3D cloud structure prediction using ResUNet and a pretrained ViT model.

Critical strengths

The dataset covers the entire year of 2025 and contains up to 61,487 samples, providing cloud types, cloud heights, and various cloud microphysical properties. It offers a valuable data resource for 3D cloud reconstruction and related machine learning research.

Critical weaknesses
  1. Insufficient validation of spatial alignment. Different viewing angles of MTG and EarthCARE may cause misalignment, especially for high-altitude clouds. The paper limits geographical coverage but provides no explicit correction or quantitative error analysis, raising concerns about label accuracy.
  2. Insufficient experimental validation. The 2-, 3-, and 4-minute variants differ in both temporal offsets and data size/composition, making it difficult to isolate the effect of temporal alignment. Moreover, results are based on single runs, and Table 4 omits the F1 score for one class and the overall mIoU.
Questions for authors
  1. How large are the spatial alignment errors between the two satellites? Have the authors evaluated positional discrepancies at different cloud altitudes and viewing angles? In particular, do these discrepancies significantly affect model training and evaluation for high-altitude clouds?
  2. How can the effects of temporal alignment accuracy and training data size be distinguished? In addition, please report the complete F1 scores for all eight classes and the overall mIoU to strengthen the reliability of the experimental conclusions.
1 comment

We thank the reviewer for the valuable comments and for highlighting the importance of spatial and temporal alignment in the construction of EarthCAREBench.

Spatial alignment and high-altitude clouds.

The dominant spatial mismatch source is cloud parallax in the FCI viewing geometry: displacement ≈h⋅tan⁡(θVZA)\approx h \cdot \tan(\theta_\text{VZA}), growing with cloud height hh and viewing zenith angle θVZA\theta_\text{VZA}. The MTG observation domain is restricted to ±45°\pm45° around the sub-satellite point to limit this effect (Section 3). Expected displacement is small for low clouds ($\approx$2–4 km for 1.5 km tops) but reaches $\approx$13–15 km for high clouds (10–12 km tops) at domain edges. As shown in Appendix B Fig. 1, most samples lie close to the sub-satellite point where displacements are smaller. We will state this limitation more explicitly in the revised manuscript.

Parallax does not appear to prevent learning of high clouds. Ice clouds — the class most exposed to parallax — achieve the highest F1 among all cloud classes (0.574–0.586 in the updated results). The most difficult classes are low liquid and mixed-phase clouds, which are less affected by parallax but frequently obscured by overlying layers. No systematic degradation with altitude is observed in per-level segmentation analysis.

Temporal alignment versus training data size.

The evaluation is controlled: all models are tested on the same 2-minute held-out split (Section 4.3), so observed differences reflect training conditions only. Within $\leq$4-minute windows, temporal mismatch has at most a minor effect. Per-scene segmentation loss shows no systematic relationship with the MTG–EarthCARE time offset. For regression, the worst 5% of cases show a median time offset only $\approx$15–25 s higher than the rest for liquid properties, but these cases also have substantially larger liquid fractions (median LWC positive fraction 0.061 vs. 0.011), so temporal mismatch is confounded with scene content.

Updated results show segmentation improves only slightly with colocation window (macro F1: 0.563 / 0.570 / 0.576; mIoU: 0.440 / 0.447 / 0.452 for 2/3/4 min), and liquid-cloud F1 — the class most sensitive to cloud evolution — is essentially unchanged (0.329 / 0.326 / 0.332). Regression shows the opposite trend: mean MAE increases slightly (0.0175 / 0.0177 / 0.0184), driven by liquid properties (LWC MAE: 0.0120 / 0.0123 / 0.0137). Differences of this magnitude cannot be reliably attributed to either temporal mismatch or sample size from single runs.

Complete per-class F1 and mIoU.

Train Δt\Delta tClearIceLiquidSupercooledIce+SCTropo. aer.Strato. aer.OtherMacro F1mIoUSSIMPSNR (dB)
2 min0.9280.5740.3290.1940.2440.7130.6550.8680.5630.4400.91030.30
3 min0.9310.5820.3260.1870.2700.7280.6530.8790.5700.4470.91830.59
4 min0.9320.5860.3320.2080.2750.7250.6780.8730.5760.4520.91430.60
The revised manuscript will include this table; per-class IoU will be added to the Appendix.

For the camera-ready version we will: (i) run a matched-size ablation subsampling each colocation window to the 2-minute training set size; (ii) train all configurations with 3 random seeds and report mean ±\pm standard deviation; and (iii) use per-pixel time offsets (stored with every sample) to study the effect of temporal misalignment directly.

Reviewer_WuFcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EarthCAREBench, a dataset pairing MTG/FCI satellite imagery with EarthCARE observations for vertically resolved atmospheric classification and cloud microphysical retrieval. The dataset covers 2025 and provides three temporal colocation windows (2, 3, and 4 minutes). The authors establish ResUNet and pretrained ViT baselines and examine how the colocation window affects retrieval performance. The dataset could be a useful resource for the community. However, several aspects of the experimental design, label quality, and evaluation need further validation before the main conclusions can be made.

Critical strengths
  1. The dataset combines recent passive and active satellite observations and covers both atmospheric classification and cloud microphysical retrieval.
  2. The authors consider relevant challenges, including temporal misalignment, sparse supervision, and severe class imbalance.
  3. The dataset format, temporal splits, and baseline models provide a useful starting point for future research.
Critical weaknesses
  1. Location: Page 5, Section 3.2; Page 7, Table 3; Pages 8–9, Evaluation and Table 4; Page 21, Table 10. The authors use a common testing set to isolate the effect of the colocation window. However, the three training datasets are constructed independently, with different sample sizes (21,259, 31,329, and 41,193), and their data distributions are not shown to be comparable. All results are from single runs. The 3-minute model achieves the highest PSNR (29.18 dB), but its SSIM is lower than that of the 4-minute model. Table 10 also reports identical mean detection F1 and non-zero RMSE for the 3- and 4-minute models at the reported precision. These results do not establish that 3 minutes provides the best trade-off or isolate the effect of temporal misalignment.

  2. Page 4, Colocation; Page 17, Appendix C.1, Figures 2 and 3. The authors acknowledge residual parallax errors and approximate FCI pixel acquisition times using scan-chunk midpoints, but neither the residual spatial displacement nor the timing error is quantified. Figure 2 uses an 8-minute window, outside the 2–4-minute range adopted in the dataset. Figure 3 reports temporal offsets and profile counts but does not assess image-label consistency. The extent of label mismatch within the actual colocation windows remains unclear.

  3. Page 8, Training and Checkpoint Selection and Evaluation; Page 9, Table 4; Page 19, Table 8; Page 21, Table 10. Approximately 92.94% of valid ice-property values and 97.93% of liquid-property values are zero, yet checkpoints are selected using validation PSNR, which is strongly influenced by these zeros. Although detection F1 and non-zero RMSE are introduced, Table 10 reports them only for ResUNet, not ViT. Table 4 also omits the "other" class F1, macro F1, mIoU, and per-class IoU. The claim below Table 10 that the SatMAE-based model better recovers microphysical magnitudes and fine-scale structures lacks corresponding quantitative evidence.

  4. Page 2, Sparse Supervision; Page 6, Task Definitions; Page 8, Evaluation; Page 9, Future Validation Plans. The models predict dense 3D fields over 128 × 128 patches, but reference labels cover only about 130 horizontal pixels per patch, and evaluation is restricted to valid along-track locations. No independent validation is provided for the horizontal structure outside the EarthCARE track. Airborne observations and radiative-transfer closure are mentioned only as future work. The spatial extent, connectivity, and consistency of the full 3D predictions therefore remain unverified.

  5. Page 2, Benchmark Contributions; Page 7, Section 4.3, Data Splits; Page 8, Evaluation. The authors describe the temporal split as "leakage-free" because training and testing do not share calendar dates. However, there is no buffer between training and validation (days 16/17) or between the end-of-month test period and the following month's training period. The paper does not report checks for overpasses crossing midnight or correlations between samples near split boundaries. The claim of leakage-free evaluation needs stronger evidence.

  6. Page 7, Pretrained ViT; Page 9, Table 4 and discussion of pretraining results. The authors compare a ResUNet trained from scratch with an MAE-pretrained ViT and attribute the improvement partly to self-supervised pretraining. Without a ViT trained from scratch, the effects of architecture and pretraining cannot be separated. The dates, size, and potential overlap of the pretraining data with test imagery are also insufficiently documented. Deferring the full pretraining description and evaluation to future work limits reproducibility and weakens this conclusion.

  7. Page 4, Quality Control; Page 7, Supervision and Regression Loss; Page 8, Evaluation; Pages 15–16, Tables 6 and 7. The authors retain quality flags but do not specify which quality-status categories are accepted or whether "valid" simply means non-missing. Several preprocessing details are also missing, including the parameter (s) in (\log(1+y/s)), the handling of test values outside the training-set scaling range, and the treatment of invalid labels in SSIM calculations. Detection F1 also uses a threshold of 0.1 in normalized units, making its interpretation dependent on preprocessing. These details are needed for reproducibility.

  8. Page 8, Section 4.4; Page 9, Table 4; Page 21, Table 10. The authors note that aerosol classes can largely be predicted from their altitude distributions and that microphysical metrics are heavily affected by zero-valued targets. However, only ResUNet and pretrained ViT baselines are reported. Simple alternatives, such as altitude-dependent class priors, climatological predictions without FCI imagery, and all-zero regression baselines, are missing. Without these comparisons, the actual benefit of the satellite imagery is difficult to assess.

  9. Page 19, Appendix C.3, Table 8; Page 7, Supervision. Table 8 reports unusually large maximum values among valid non-zero retrievals. For example, the maximum ice water content is 1.274242 kg/m³, compared with a non-zero mean of 0.000087 kg/m³. The maximum ice effective radius is 0.005861 m, compared with a mean of 0.000044 m. The authors should clarify the origin, frequency, and quality status of these extreme values and whether they affect the training labels or scaling statistics.

  10. Page 1, Abstract. The abstract describes the dataset in detail, including temporal windows, geographical coverage, and patch size, but does not report any main experimental findings. It would benefit from a shorter background and a clearer summary of the key contributions and benchmark results.

Questions for authors
  1. Can the authors control for training sample size and composition when comparing colocation windows, and report variability across multiple runs?

  2. Can the authors quantify image-label misalignment within the actual 2-, 3-, and 4-minute windows?

  3. Can the missing benchmark metrics, including ViT detection F1 and non-zero RMSE, be provided?

  4. Is there any independent evidence supporting the accuracy of the predicted 3D fields outside the EarthCARE track?

  5. Can the authors provide a ViT-from-scratch comparison, simple statistical baselines, and the missing preprocessing and data quality filtering details?

1 comment

We thank the reviewer for the thorough and constructive review. Below we address each question, folding in the related weaknesses, and then respond to the remaining points. We indicate what is already in the manuscript, what we add in this response, and what will be added for the camera-ready version.

  1. Controlling for training sample size and composition; variability across runs. We agree that sample size and composition are confounded with the colocation window. The evaluation is controlled: all models are tested on the same held-out 2-minute split using a shared calendar-day partition (Section 4.3). The training sets differ in size (21,259 / 31,329 / 41,193 samples for 2-, 3-, and 4-minute windows; Table 3) and scene composition, which is intentional — the goal is to study the trade-off between colocation quality and training data quantity. Segmentation improves slightly with larger windows (macro F1: 0.563 / 0.570 / 0.576; mIoU: 0.440 / 0.447 / 0.452), while regression mean MAE degrades slightly (0.0175 / 0.0177 / 0.0184).

For the camera-ready version we will (i) subsample the 3- and 4-minute training sets to match the 2-minute size to isolate the colocation effect, (ii) report 5- and 6-minute variants to show the performance turning point, (iii) report training-set statistics for each variant, and (iv) train each configuration with 3 random seeds and report mean ± standard deviation.

  1. Quantifying image-label misalignment within the 2–4-minute windows. Cloud advection at 10–30 m/s produces at most 1–7 FCI pixel displacements within a 4-minute window. A qualitative analysis with expert review of 200 samples per window found obvious discrepancies only beyond 7–8 minutes. Quantitatively, we find no systematic relationship between per-scene segmentation loss and time offset. For regression, the worst 5% of cases show a median time difference only ≈15–25 s higher than the rest for liquid properties, confounded with scene content. Figure 2 uses an 8-minute window to illustrate the mechanism; a reliable spatial quantification within the training windows is not possible since EarthCARE provides only a 1D curtain against which the full 2D FCI scene cannot be evaluated.

In the revised manuscript we will include example plots for the 2–4-minute range and provide an indirect quantitative assessment based on FCI–EarthCARE MSI cloud-occurrence agreement per time-offset bin.

  1. Missing benchmark metrics; checkpoint selection. PSNR is not used as the sole selection criterion. We retain checkpoints selected by lowest validation loss and highest mIoU, F1, PSNR, and SSIM. The table below shows that segmentation results differ by at most 0.003 in macro F1 and mIoU between the highest-PSNR and best-cloud-F1 checkpoints, confirming the conclusions are not sensitive to this choice:

Train Δt Criterion Macro F1 mIoU 2 min highest PSNR (ep50) 0.563 0.440 2 min best cloud F1 (ep73) 0.566 0.443 3 min highest PSNR (ep93) 0.570 0.447 3 min best cloud F1 (ep80) 0.572 0.449 4 min highest PSNR (ep46) 0.576 0.452 4 min best cloud F1 (ep58) 0.578 0.453

In the revised manuscript: (i) Table 10 will add detection F1 and non-zero-target RMSE for all models; (ii) Table 4 will report complete per-class F1 for all eight classes with macro F1 and mIoU; and (iii) checkpoint selection will switch to a zero-insensitive criterion, with PSNR results kept for reference.

  1. Independent evidence outside the EarthCARE track. All reported metrics are computed along the EarthCARE track, the only location with reference profiles. Two independent validations are planned: comparison with airborne observations from the PERCUSION/ORCESTRA campaign and 3D radiative-transfer closure against EarthCARE radiances and fluxes (Section 5).

  2. ViT from scratch and baselines. For the camera-ready version we will add a ViT trained from random initialisation on the same data and splits to separate architecture from pretraining effects.

Other points.

Data split: We will add one-day buffers at both training/validation and validation/test boundaries, describe the split as calendar-day separated, and report results on the re-split data. Extreme values: The maxima (IWC 1.27 kg m⁻³; ice effective radius 5.9 mm) are most likely retrieval outliers from the EarthCARE L2 product. We will report their frequency, exclude values above literature-based bounds, and apply robust-percentile min–max scaling. The effect on the benchmark will be reported. Valid pixels and quality flags: Valid pixels are defined as non-missing; quality flags are not currently applied but an analysis was conducted to identify systematic issues. Full preprocessing details will be released with the dataset and code.

All results, splits, and code will be released to allow full reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.