- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a dataset which focuses on understanding the 3D structure of clouds. Observations from two sensors are combined, one which provides dense data without explicit 3D structure and another which provides sparse 3D structure. The idea would be to use the dense data to predict the sparse samples and thus be able to generate a dense wall-to-wall map of estimated 3D structure. A baseline U-Net across a few settings and more limited results from a ViT model (with MAE pretraining) are presented.
Critical strengths
- The paper is well written.
- The goal of better understanding climate drivers is noble.
- The selection of baseline methods is sensible.
- The data split strategy seems sensible given the nature of the data.
Critical weaknesses
Summary:
This is going to be a short review, because I honestly don't think much of the content of the paper is really relevant to the ICLR community. So, I'm going to focus on how I think about the value of a dataset paper. First, it should be well explained and as open as possible. This paper does that. Second, it should ideally have some significant potential value as a signifier of progress in data interpretation (e.g., strong performance on this dataset should have some real-world impact, either directly on this problem or the indirectly on other problems). This is a little weaker, because we can't really see how performance improvements on the proposed metrics will impact "estimates of climate sensitivity". Third, it should motivate the development of novel methods in representation learning or provide an interesting evaluation case that will help us better understand existing methods. This is the weakest element of this work. The paper does not articulate this and I don't see any obvious ways in which this is likely to lead to novel methods in representation learning. I can see it being a good target for hyperparameter tuning of existing methods, but I don't think there will be impact outside the remote sensing communit.
Specific weaknesses:
- This seems more like a remote sensing paper than an ICLR paper. It's is unclear whether this proposed benchmark would lead to any truly novel innovation in the representation learning domain that couldn't already be handled by other similar problems in remote sensing (e.g., canopy height estimation comes to mind as a problem that has a similar sparse-training dense-inference setup, but I'm sure there are many). This is evidenced by the citations, which are dominated by remote sensing papers with very little that indicates the recent work in representation learning that could be impacted by the release of this dataset.
- The evaluation metrics are sensible from an ML perspective, but they don't seem to connect to the task in a meaningful way. A claimed motivation for this in the climate science domain is that it will help resolve some ambiguities in the inputs to current models, but nothing about the evaluation really tests the impact of this.
- It's unclear how meaningful the improvements in metrics across different baseline settings are relative to just using the sparse labels and dense labels directly as inputs to a simulation model.
- Stated differently: Are the PSNR/SSIM/F1 numbers good? Are they bad? How far are we from having estimates that would be useful?
- One additional concern is that the experiments appear to be incomplete. For example, the unet is evaluated at three temporal offsets, but the ViT model is only evaluated at one.
Questions for authors
Are there other dataset papers that have been published at ICLR that you think are similar in nature to this paper? How would compare the potential and actual impact on the ICLR community of a few of these previous papers to this paper?
We thank the reviewer for the valuable comments and for highlighting the importance of spatial and temporal alignment in the construction of EarthCAREBench.
Spatial alignment and high-altitude clouds.
The dominant spatial mismatch source is cloud parallax in the FCI viewing geometry: displacement ≈h⋅tan(θVZA), growing with cloud height h and viewing zenith angle θVZA. The MTG observation domain is restricted to ±45° around the sub-satellite point to limit this effect (Section 3). Expected displacement is small for low clouds ($\approx$2–4 km for 1.5 km tops) but reaches $\approx$13–15 km for high clouds (10–12 km tops) at domain edges. As shown in Appendix B Fig. 1, most samples lie close to the sub-satellite point where displacements are smaller. We will state this limitation more explicitly in the revised manuscript.
Parallax does not appear to prevent learning of high clouds. Ice clouds — the class most exposed to parallax — achieve the highest F1 among all cloud classes (0.574–0.586 in the updated results). The most difficult classes are low liquid and mixed-phase clouds, which are less affected by parallax but frequently obscured by overlying layers. No systematic degradation with altitude is observed in per-level segmentation analysis.
Temporal alignment versus training data size.
The evaluation is controlled: all models are tested on the same 2-minute held-out split (Section 4.3), so observed differences reflect training conditions only. Within $\leq$4-minute windows, temporal mismatch has at most a minor effect. Per-scene segmentation loss shows no systematic relationship with the MTG–EarthCARE time offset. For regression, the worst 5% of cases show a median time offset only $\approx$15–25 s higher than the rest for liquid properties, but these cases also have substantially larger liquid fractions (median LWC positive fraction 0.061 vs. 0.011), so temporal mismatch is confounded with scene content.
Updated results show segmentation improves only slightly with colocation window (macro F1: 0.563 / 0.570 / 0.576; mIoU: 0.440 / 0.447 / 0.452 for 2/3/4 min), and liquid-cloud F1 — the class most sensitive to cloud evolution — is essentially unchanged (0.329 / 0.326 / 0.332). Regression shows the opposite trend: mean MAE increases slightly (0.0175 / 0.0177 / 0.0184), driven by liquid properties (LWC MAE: 0.0120 / 0.0123 / 0.0137). Differences of this magnitude cannot be reliably attributed to either temporal mismatch or sample size from single runs.
Complete per-class F1 and mIoU.
For the camera-ready version we will: (i) run a matched-size ablation subsampling each colocation window to the 2-minute training set size; (ii) train all configurations with 3 random seeds and report mean ± standard deviation; and (iii) use per-pixel time offsets (stored with every sample) to study the effect of temporal misalignment directly.