acceptodds
Withdrawn from ICLR 2027

Less Is More: Semantic Curation of Earth Observation Data for Self-Supervised Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Earth-observation (EO) foundation models are pretrained on massive satellite archives that contain substantial spatial redundancy. Yet EO pretraining data are rarely curated semantically, and are instead typically used as collected or sampled using hand-designed geographic and metadata-based rules. We investigate the impact of data curation on self-supervised EO pretraining and introduce GeoSelect, a training-free semantic curation pipeline tailored to EO. Starting from embeddings of a frozen EO foundation model, GeoSelect removes acquisition-induced variability before selecting a fixed-size subset of diverse observations. On the globally distributed Sentinel-2 Major-TOM archive, pretraining on only 10% of the data yields better representations than pretraining on the full archive. Across GEO-Bench, this improves linear-probing performance by +2.8 points in classification and +5.5 mIoU in segmentation. GeoSelect also outperforms geographic, metadata-based, and standard semantic curation baselines. These gains are robust across curation budgets, and the approach extends across sensing modalities. Our results show that semantic curation can improve EO representation quality while reducing the amount of pretraining data by an order of magnitude. We will release the curation pipeline and curated subsets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
data curation, self-supervised learning, Earth observation, remote sensing, foundation models, dataset pruning, data-efficient pretraining, semantic deduplication

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZWrgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GeoSelect to improve data curation processes for large-scale remote sensing self-supervised pre-training. In particular, GeoSelect processes embeddings extracted from an existing foundation model by (1) correcting acquisition-induced variability and (2) applying spatial regularization. Based on these processed embeddings, a fixed number of representative observations is then selected through semantic deduplication. Downstream performance on GeoBench for 5 classification and 5 segmentation tasks show improvements over existing sampling methods from the Major-TOM pre-training dataset when fixing the scale of reduction at 10%.

Critical strengths
  • Relevant and interesting research question: improving the quality of RS pretraining datasets while reducing its size, especially in the light of needs for computational efficiency.
  • Interesting methodological approach that aims to distinguish meaningful semantic from acquisition diversity when selecting RS pre-training datasets.
  • Evaluation across a larger group of classification and segmentation tasks, as well as for S1 downstream tasks.
Critical weaknesses
  • “No ocean“ run neither described in the paper nor used in a fair comparison - Please describe it and compare with “Spatial Regul. + SDD (no ocean)“ and “Acq. correc. + SDD (no ocean)“; the results without “No ocean“ are not fully in favor of GeoSelect
  • Only one seed used - The differences of the main results in Table 1 are small; I would expect at least three independent seeds per run to demonstrate that GeoSelect is consistently outperforming other methods.
  • Limited generality of GeoSelect - Please show the benefits on at least one more S2 pre-training dataset (e.g., SSL4EO-S12 [1]).
  • Missing comparison with another relevant sampling strategy: the strategy introduced by Wang et al. [1] for constructing SSL4EO-S12 (which builds upon and improves the sampling strategy of Mañas et al. [2])
  • Figure 5 does not consider all stronger methods - I would like to see a comparison of smaller percentages than 10% of the pre-training dataset for (1) random subset baseline, (2) Soil stratified, (3) UniverSat Hier. k-means, (4) UniverSat pHash + Hier. k-m., (5) Spatial Regul. + SDD, (6) Acq. correc. + SDD
  • The introduction does not sufficiently well explain GeoSelect (particularly L045-L050) ; Which limitations of existing methods does GeoSelet address, which components are novel, which build on existing techniques?

Moderate weaknesses

  • Linear probing still involves hyperparamter tuning for evaluation - I would like to see kNN results for the classification tasks. I am aware that fine-tuning on S2 datasets can lead to very similar results and does not proof representations quality well, therefore I am okay with leaving that experiment out, but kNN is needed.
  • The paper does not contain an explicit limitation section
  • Inference time to retrieve feature via UniverSat not stated
  • Contrastive-like SSL algorithms not considered (e.g., MoCoV2, DINO)


Minor weaknesses

  • UniverSat not cited in the first Figure
  • When referring to the supplemental material, better explicitly ref the Figures, Tables or Section
  • Figure 1 is not really informative
  • Linear Probing hyperparameters are missing in experimental details in supplemental material

Formatting

  • L014: Line break in abstract; L057: „Contributions“ => our contributions are; References like „Fig. 9“, but its written „Figure 9“ this is inconsistent; Figure 1 never referenced

References

[1] Y. Wang et al. "SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation [Software and Data Sets]." IEEE Geoscience and Remote Sensing Magazine 11.3 (2023): 98-106.

[2] O. Manas et al. "Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data.“ IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

Questions for authors
  • L136: The author claim that season variability does not reflect meaningful differences. I might disagree. Please justify!
  • Burgert et al. [3] show that downstream performance can saturate for pre-training on S2-based dataset at around 100k and 200k samples, how can we distinguish the benefits of the proposed selection strategy from the effects of simply reducing a highly redundant dataset to a more appropriate size?

References

[3] T. Burgert et al. "Rank-based Geographical Regularization: Revisiting Contrastive Self-Supervised Learning for Multispectral Remote Sensing Imagery." IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2026.

Reviewer_Av7LOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at data selection for self-supervised learning in Earth observation. Its method, GeoSelect, uses embeddings from a frozen model to pick a subset of a satellite archive. It corrects feature shifts caused by acquisition conditions, applies spatial regularization, and removes semantically redundant samples. On Major-TOM, MAE models pretrained on the selected subsets beat full-archive pretraining on GEO-Bench at the same training compute. The paper also varies the subset size and tests whether a selection made on optical imagery carries over to radar pretraining.

Critical strengths
  1. The problem is practical. Large satellite archives are highly redundant, so better selection could cut storage and data-transfer costs.
  2. The baselines and tasks are broad. The comparison includes geographic, metadata-based, and semantic selection methods. It covers both classification and segmentation, and the segmentation gains over full-archive pretraining are large.
  3. The acquisition analysis motivates the method well. Figures 2, 7, and 9 show that acquisition groups explain a structured part of the variation in the embeddings. Each MAE model is also trained from scratch, which keeps the effect of data selection separate from representation learning.
Critical weaknesses
  1. The ablations don’t clearly support the main contribution. In Table 1, spatial regularization with SemDeDup improves classification by 2.6 points and segmentation by 4.9 mIoU. Adding acquisition correction lowers the classification gain to 2.1 points and raises the segmentation gain only to 5.2 mIoU. No repeated runs are reported. The results therefore don’t show a reliable benefit from adding acquisition correction to the simpler method.
  2. The best results rely on an extra ocean-filtering rule. The filter raises the classification gain from 2.1 to 2.8 points and the segmentation gain from 5.2 to 5.5 mIoU. Table 1 doesn’t apply the same filter to spatial regularization alone, standard SemDeDup, or random selection. So it is unclear how much of the final gain actually needs the full method.
  3. The matched training budget is hard to verify. Section 4 says all methods use the same number of optimizer steps. Table 4, however, lists 200 epochs, early stopping, and an evaluation interval defined in epochs. The paper doesn’t explain how the step count stays matched across datasets of different sizes under these settings. It is also unclear which signals drive learning-rate scheduling and checkpoint selection.
  4. The evidence that acquisition effects are removed is indirect. Pass and block identities are tied to geography, so their estimated offsets may include real differences between scenes. Lower feature variation between neighboring locations doesn’t by itself show that semantic information survives. A direct test is missing: one showing that the correction removes acquisition information while keeping scene information.
Questions for authors
  1. How is an epoch defined for subsets of different sizes? What are the actual optimizer step counts for the full archive and the main subsets? Which metrics drive early stopping, learning-rate scheduling, and checkpoint selection?
  2. Why does the full method do worse on classification than spatial regularization alone? Does the small segmentation gain hold across several training seeds?
  3. How do spatial regularization alone, standard SemDeDup, and random selection perform with the same ocean filter applied? How are ocean samples identified?
  4. Is there direct evidence that acquisition correction preserves semantic differences? A small evaluation on repeated observations of the same locations could help separate acquisition effects from geographic content.
Reviewer_vQDVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Investigate the impact of data curation on self-supervised EO pretraining, showing that semantic curation can improve EO representation quality while reducing the amount of pretraining data by an order of magnitude
  • Introduced GEOSELECT, a training-free semantic curation pipeline tailored to EO: GEOSELECT removes acquisition-induced variability before selecting a fixed-size subset of diverse observations, and pretraining on only 10% of the data yields better representations. Across GEO-Bench, this improves linear-probing performance by +2.8 points in classification and +5.5 mIoU in segmentation
  • GEOSELECT also outperforms geographic, metadata-based, and standard semantic curation baselines. These gains are robust across curation budgets, and the approach extends across sensing modalities
Critical strengths
  • The paper is well-written and easy to follow
  • The proposed training-free semantic curation pipeline (GEOSELECT) yields better representations than pretraining on the full archive and also outperforms geographic, metadata-based, and standard semantic curation baselines
  • The proposed GEOSELECT also extends across sensing modalities (Sentinel-2 to Sentinel-1)
  • Figures 2 and 3 provide interesting visualizations for the proposed pipeline and show evidence to support the design rationale
Critical weaknesses
  • "We represent each observation using a pretrained EO foundation model ϕ" -> Would the chosen pretrained EO foundation model affect the experiment results? Why do we need to use a pretrained EO foundation model to semantically curate the dataset, and then train another SSL model? Can't we just do distillation?
  • "For curation, however, it is undesirable: acquisition effects can make otherwise similar samples appear artificially diverse, causing a selection method to spend part of its budget covering acquisition variability rather than scene diversity." -> Why is acquisition variability not important? For tasks like time series modelling or change detection, this can have a huge effect on downstream performance
  • Would the distribution of pretrained data and evaluation set affect the results? For example, using GEOSELECT, we select the overrepresented samples (close to the evaluation set) in the pretrained data and therefore gain higher performance?
Questions for authors
  1. "When the output of ϕ is not already normalized, we ℓ2-normalize the embeddings to the unit sphere." -> Ablation study on the normalization effects?
  2. "These factors are consequently also encoded by ϕ: observations of similar scenes may lie far apart in feature space simply because they were acquired under different conditions." -> Could you provide evidence to support this claim?
  3. Provide more details about the MajorTOM dataset
  4. Why don't you include DINOv3 for the baseline experiments?
  5. The experiment results make me think of another paper, "How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Models" (https://arxiv.org/abs/2601.22841). Do you think your paper also supports the conclusion made in this paper?
  6. "Figure 4: Selected samples. We consider samples retained by GEOSELECT at a 10% budget and restrict them to two land-cover groups: > 80% forest and > 50% urban cover." Why did you choose these two land-cover groups? reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.