- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article presents a new open-vocabulary dataset for remote sensing imagery. It mixes RGB images from aerial data sources over 3 countries with high-resolution SAR data. Human annotators have labeled the images at the pixel-level to produce ~2800 RGB images semantic masks pairs, about half of which are also co-registered with SAR. The dataset cover a moderately diversified taxonomy of 44 classes. Baseline evaluations are conducted on this dataset by evaluating off-the-shelf zero-shot open vocabulary segmenters and by training existing open vocabulary segmenters on it.
Critical strengths
-
Novel dataset for open-vocabulary segmentation of remote sensing imagery, that includes two modalities (optical and SAR image), a setup that is under-explored in the literature.
-
Moderately large dataset but that includes a significant amount of human-annotated semantic masks.
Critical weaknesses
A. Pretraining dataset or benchmark dataset?
In the open vocabulary semantic segmentation literature, there is a clear distinction between:
- the datasets that are used to pretrain open vocabulary segmenters,
- the datasets that are used to evaluate open vocabulary segmenters (the "downstream" datasets).
Typically, papers tend to pretrain on a custom dataset (LandDiscover50k, OVRSIS, etc.) and test on a collection of semantic segmentation, including iSAID, FLAIR, LoveDA, etc. This is the protocol used to evaluate various models such as RSCLIP, ROSS, AeroSeg, but also baselines that the authors have included in their own work, such as SegEarth-OV.
In comparison, the proposed OpenEarth-OVS44 dataset is used both to train and evaluate models (e.g. Table 4). I do not think this gives a complete understanding of the capabilities of the models, as the construction of the dataset makes it less challenging than the usual protocol. This is not standard and I do not really understand why the authors did not follow experimental setups from previous works, which could have allowed for a cleaner comparison between the methods, while also disentangling the impact of changing the training dataset.
In the end, I think the question is as follows: do you want other people to use this dataset as a pretraining dataset or a benchmark? The title and experiments tend to suggest it is the former, but in that case then the evaluation has to be conducted on downstream tasks, similarly to what has been done in previous works, e.g. OVRSISBenchV2 & others.
B. Comparison to existing datasets
In addition, the paper seems to imply in section 2 that the field is lacking open vocabulary datasets with a significant number of classes:
They fall into two regimes: unimodal RGB benchmarks (Chaudhuri et al., 2018; Waqas Zamir et al., 2019) with up to 17 classes (l.157)
This is inaccurate. It is true that most existing open vocabulary benchmarks aggregate existing classical datasets. However, there are benchmarks that are significantly larger and with many more classes than what is alluded to in the paper. For example, the paper ignores:
- OVRSIS95k (95k images, 35 classes)
- PanoEarth (150k images, 111 classes)
- LandDiscover50k (51k images, 40 classes)
These datasets are bigger and have a similar number of classes (or more classes) than the proposed OpenEarth-OVS44. I acknowledge that the proposed dataset still fills a gap: it contains multimodal SAR/optical pairs, which few prior works propose. However this means that the OpenEarth-OVS44-Optical experiments are very incomplete: they should also compare to models pretrained
C. Missing baselines
The experimental results also pick a set of open vocabulary segmenters from the literature, but several notable ones are missing without any clear reason: RSCLIP, ROSS, AeroSeg, OVRS, to name a few. I am not a stickler for extensive comparison over all existing baselines, but at least explaining why you choose these baselines would make for a more convincing paper.
Questions for authors
Major questions
- Is this dataset supposed to be for pretraining or for benchmarking? If it is pretraining, why not evaluate on downstream datasets for fairer comparison? If is benchmarking, why compare models trained on it?
- What is the positioning of this dataset against the datasets mentioned in critical weakness B?
- What is the motivation between the choice of baselines?
Minor questions
- 3.2.3.: it's unclear how the train/val/test sets were sampled and if any measure was taken to avoid data leakage, e.g. due to spatial autocorrelation.
- Table 1: how were these classes chosen? They seem very close to the classes that are commonly found in iSAID, fMoW, FLAIR and other datasets, and not particularly fine-grained.
- Table 3: what is the influence of this split?
- Table 4: why are SOTA methods so underperforming? Is there a reason to believe this dataset is particularly hard?
- What is the annotation quality, i.e. annotator agreement and/or comparison to a gold standard?
Suggestions/remarks
- What does "mainly" mean in "the SAR imagery mainly provides amplitude"? (l.221) Either the SAR imagery contains phase information or it does not.
- It's not clear on what criteria the images have been curated. (l.226) It's also unclear what is the effect of this curation, maybe the less-well curated images are still useful to train models. An experiment with/without curation would have been helpful here.
- 3.2.2: I recommend releasing the annotations from the multiple annotators, especially when they disagree, as human uncertainty has been shown to improve robustness of deep nets.
- Figure 4: please use a barplot, a lineplot is not suitable for this kind of non-sequential data.