acceptodds
Withdrawn from ICLR 2027

OpenEarth-OVS44: A Multimodal Remote Sensing Dataset for Open-Vocabulary Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In remote sensing, open-vocabulary segmentation is particularly important for flexible and scalable land-cover understanding across diverse regions, as it improves generalization to novel categories without requiring exhaustive pixel-level annotations. Despite its great promise, progress in this area remains largely constrained by existing datasets with limited category coverage and overly coarse taxonomies, and therefore struggles to support fine-grained semantic understanding required in real-world applications such as urban functional zoning, carbon accounting, and post-disaster assessment. To address these challenges, we present OpenEarth-OVS44, a multimodal benchmark with 290,587 expert-verified segments from 2,832 RGB images and 1,341 strictly registered RGB-SAR pairs at sub-meter spatial resolution, covering 40 cities across France, Japan, and the United States under a unified 44-class taxonomy organized into 9 base categories. Three evaluation tracks are provided: an RGB-only fine-grained benchmark, a multimodal track with diverse synthetic cloudy conditions for weather-robust evaluation, and a held-out cross-region track. Extensive experiments on representative open-vocabulary segmentation models demonstrate that training on OpenEarth-OVS44 substantially improves fine-grained segmentation performance over the strongest training-free baseline, with improvements of up to 7.1 mIoU points on unseen categories and 27.6 mIoU points on seen categories. The dataset and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Remote Sensing Dataset, Open-Vocabulary Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iGCpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This article presents a new open-vocabulary dataset for remote sensing imagery. It mixes RGB images from aerial data sources over 3 countries with high-resolution SAR data. Human annotators have labeled the images at the pixel-level to produce ~2800 RGB images semantic masks pairs, about half of which are also co-registered with SAR. The dataset cover a moderately diversified taxonomy of 44 classes. Baseline evaluations are conducted on this dataset by evaluating off-the-shelf zero-shot open vocabulary segmenters and by training existing open vocabulary segmenters on it.

Critical strengths
  • Novel dataset for open-vocabulary segmentation of remote sensing imagery, that includes two modalities (optical and SAR image), a setup that is under-explored in the literature.

  • Moderately large dataset but that includes a significant amount of human-annotated semantic masks.

Critical weaknesses

A. Pretraining dataset or benchmark dataset?

In the open vocabulary semantic segmentation literature, there is a clear distinction between:

  • the datasets that are used to pretrain open vocabulary segmenters,
  • the datasets that are used to evaluate open vocabulary segmenters (the "downstream" datasets).

Typically, papers tend to pretrain on a custom dataset (LandDiscover50k, OVRSIS, etc.) and test on a collection of semantic segmentation, including iSAID, FLAIR, LoveDA, etc. This is the protocol used to evaluate various models such as RSCLIP, ROSS, AeroSeg, but also baselines that the authors have included in their own work, such as SegEarth-OV.

In comparison, the proposed OpenEarth-OVS44 dataset is used both to train and evaluate models (e.g. Table 4). I do not think this gives a complete understanding of the capabilities of the models, as the construction of the dataset makes it less challenging than the usual protocol. This is not standard and I do not really understand why the authors did not follow experimental setups from previous works, which could have allowed for a cleaner comparison between the methods, while also disentangling the impact of changing the training dataset.

In the end, I think the question is as follows: do you want other people to use this dataset as a pretraining dataset or a benchmark? The title and experiments tend to suggest it is the former, but in that case then the evaluation has to be conducted on downstream tasks, similarly to what has been done in previous works, e.g. OVRSISBenchV2 & others.

B. Comparison to existing datasets

In addition, the paper seems to imply in section 2 that the field is lacking open vocabulary datasets with a significant number of classes:

They fall into two regimes: unimodal RGB benchmarks (Chaudhuri et al., 2018; Waqas Zamir et al., 2019) with up to 17 classes (l.157)

This is inaccurate. It is true that most existing open vocabulary benchmarks aggregate existing classical datasets. However, there are benchmarks that are significantly larger and with many more classes than what is alluded to in the paper. For example, the paper ignores:

These datasets are bigger and have a similar number of classes (or more classes) than the proposed OpenEarth-OVS44. I acknowledge that the proposed dataset still fills a gap: it contains multimodal SAR/optical pairs, which few prior works propose. However this means that the OpenEarth-OVS44-Optical experiments are very incomplete: they should also compare to models pretrained

C. Missing baselines

The experimental results also pick a set of open vocabulary segmenters from the literature, but several notable ones are missing without any clear reason: RSCLIP, ROSS, AeroSeg, OVRS, to name a few. I am not a stickler for extensive comparison over all existing baselines, but at least explaining why you choose these baselines would make for a more convincing paper.

Questions for authors

Major questions

  1. Is this dataset supposed to be for pretraining or for benchmarking? If it is pretraining, why not evaluate on downstream datasets for fairer comparison? If is benchmarking, why compare models trained on it?
  2. What is the positioning of this dataset against the datasets mentioned in critical weakness B?
  3. What is the motivation between the choice of baselines?

Minor questions

  • 3.2.3.: it's unclear how the train/val/test sets were sampled and if any measure was taken to avoid data leakage, e.g. due to spatial autocorrelation.
  • Table 1: how were these classes chosen? They seem very close to the classes that are commonly found in iSAID, fMoW, FLAIR and other datasets, and not particularly fine-grained.
  • Table 3: what is the influence of this split?
  • Table 4: why are SOTA methods so underperforming? Is there a reason to believe this dataset is particularly hard?
  • What is the annotation quality, i.e. annotator agreement and/or comparison to a gold standard?

Suggestions/remarks

  • What does "mainly" mean in "the SAR imagery mainly provides amplitude"? (l.221) Either the SAR imagery contains phase information or it does not.
  • It's not clear on what criteria the images have been curated. (l.226) It's also unclear what is the effect of this curation, maybe the less-well curated images are still useful to train models. An experiment with/without curation would have been helpful here.
  • 3.2.2: I recommend releasing the annotations from the multiple annotators, especially when they disagree, as human uncertainty has been shown to improve robustness of deep nets.
  • Figure 4: please use a barplot, a lineplot is not suitable for this kind of non-sequential data.
Reviewer_fshfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces OpenEarth-OVS44, a multimodal remote sensing benchmark for fine-grained open-vocabulary segmentation. The dataset contains 44 categories, 2,832 RGB images, and 1,341 paired RGB-SAR images. It provides three evaluation tracks covering optical segmentation, synthetic cloudy conditions, and cross-region generalization. The authors evaluate several existing OVS models and report improvements after training on the proposed dataset.

Critical strengths
  1. The dataset provides a more detailed category taxonomy than several existing remote sensing OVS benchmarks.

  2. The inclusion of paired RGB-SAR imagery and cross-region evaluation is potentially useful for future research.

Critical weaknesses
  1. The reported improvements are largely based on comparisons between models with different backbones, training protocols, and input modalities, making it difficult to attribute the gains to the proposed dataset.

  2. In Table 4, all trained ViT-B/16 models achieve lower unseen-category mIoU than the training-free SegEarth-OV baseline on both Optical and Cross-region benchmarks. This directly weakens the claim that training on the proposed dataset consistently benefits novel-category generalization.

  3. The headline improvement of 7.1 unseen mIoU points under cloudy conditions compares a trained multimodal model with a training-free RGB-only model using a different backbone. This comparison does not isolate the benefit of dataset-based training.

  4. The unseen-category split is highly concentrated: 9 of the 13 unseen categories belong to the Road and Water groups. A single predefined split is insufficient to establish general open-vocabulary evaluation reliability.

  5. The cloudy benchmark relies entirely on synthetic cloud generation without validation on real cloud-contaminated imagery, limiting the strength of the weather-robustness claims.

  6. Although the authors describe an expert annotation review process, quantitative inter-annotator agreement and detailed spatial split-integrity analyses are missing. These are particularly important for a benchmark containing visually ambiguous fine-grained categories.

Questions for authors
  1. Can the authors provide controlled comparisons with matched backbones and input modalities to demonstrate that training on OpenEarth-OVS44 improves unseen-category generalization?

  2. How robust are the benchmark results to alternative seen/unseen category splits? Does the conclusion hold when unseen categories are distributed more broadly across the taxonomy?

  3. Can the authors provide quantitative annotation agreement results, spatial split-integrity checks, and validation using real cloudy imagery?

Reviewer_3AA9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenEarth-OVS44, a benchmark for fine-grained open-vocabulary semantic segmentation in remote sensing. It contains 2,832 RGB images from 40 cities in France, the United States, and Japan, including 1,341 images with registered SAR counterparts and 290,587 annotated segments. The taxonomy comprises 44 classes organized into nine parent categories. The benchmark provides RGB-only evaluation, RGB-SAR evaluation under synthetic cloud degradation, and cross-region evaluation with Japan held out for testing. Seven representative methods are evaluated, five of which are trained on the proposed dataset. The results highlight substantial challenges in fine-grained recognition, unseen-class generalization, and segmentation under degraded optical observations.

Critical strengths
  1. The combination of a fine-grained taxonomy and paired RGB-SAR observations is practically valuable. Relative to the benchmarks listed in Table 3, the 44-class annotations enable more detailed evaluation, including distinctions between normal roads and highways, or residential buildings and shopping malls. This exposes questions that coarse land-cover benchmarks cannot adequately measure. The dataset contribution is meaningful in its own right and does not require a new segmentation architecture to be valuable.
  2. The evaluation covers complementary method families and generalization settings. The paper includes training-free approaches, natural-image OVS models, remote-sensing OVS models, and a multimodal method. Holding out Japan provides a useful geographic transfer setting. Reporting seen, unseen, and overall mIoU separately also helps reveal whether improvements extend beyond the supervised categories.
Critical weaknesses
  1. The reliability of the annotations and the independence of the benchmark splits are insufficiently substantiated. Section 3.2 describes expert verification, annotation effort, and average registration RMSE, which are useful. However, it does not provide per-class support statistics, independent reannotation agreement or error estimates, or sufficiently detailed rules for potentially overlapping or ambiguous labels.
  2. Several experimental comparisons do not isolate the factors underlying the main conclusions. The Optical and Cloudy tracks use different training/test sample counts and evaluate 44 and 43 classes, respectively. Differences between these columns therefore do not directly measure the effect of cloud degradation alone.
Questions for authors
  1. Please provide scene- or geography-based split manifests, explain how overlapping crops are excluded, and describe split consistency across the Optical and Cloudy tracks? Are unseen pixels ignored, mapped to background, or treated otherwise during training? Are unseen labels used for validation or model selection?
  2. Please provide per-class image, segment, and pixel counts, together with an independent reannotation or sampled audit? What mutually exclusive labeling rules and evidence are used for categories such as airport/terminal/runway and factory/shopping mall?
  3. Please report clear RGB, synthetic-cloud RGB, SAR-only, and RGB-SAR results on identical paired test samples, with a common class set and explicit training conditions? This would directly affect the assessment of weather robustness and the contribution of SAR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.