- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The main contribution of this paper is the proposal of the HSRest hyperspectral satellite image inpainting benchmark, which integrates standardized data partitioning, limited-data training, and cross-geographical testing into a unified evaluation protocol. This benchmark provides a reusable basis for comparing different methods under various inpainting tasks and evaluation conditions.
Critical strengths
-
Addressing the differences in data selection, degradation methods, training and testing partitioning, and evaluation metrics in existing research, this paper provides a unified data partitioning and evaluation process, contributing to improved consistency and reproducibility in method comparisons.
-
Cross-geographical experiments require the model to be trained on one geographic group and tested on other groups with non-overlapping countries and scenes. This design allows for a more direct examination of the model's performance under varying geographic distributions.
-
The paper covers tasks such as denoising, super-resolution, and image inpainting, and compares the performance of methods under different training data sizes, degradation levels, geographic partitioning, and evaluation metrics, helping to reveal performance differences that are difficult to reflect with a single experimental setup.
Critical weaknesses
- In limited-data experiments, the training sample size and geographic composition change simultaneously.
Appendix Table 10 shows that 10% of the training subset comes from Mongolia and Canada, while the 20% and 30% subsets use different combinations of countries. Therefore, performance differences under different data proportions may be influenced by the sample size, land cover composition, texture features, and acquisition conditions simultaneously, and cannot be solely attributed to the amount of training data. The authors have acknowledged this in Section 4.3 and the Limitations section, defining this setting as a robustness test under limited data conditions. It is recommended to further add control experiments that maintain similar country composition and proportions, only changing the sample size, to more clearly analyze the impact of data scale. Furthermore, Table 10 summarizes the 20% subset into 3 countries and 3,243 tiles, but the column explicitly labelling the 20% of countries and tile counts is inconsistent with the summarized values; the authors are advised to verify and clarify this.
- Details regarding the input construction and task adaptation of some baseline models still need clarification.
For example, Fusformer originally used low-resolution hyperspectral images and high-resolution multispectral images as input, while the super-resolution degradation protocol in this paper primarily describes the downsampling process of hyperspectral images. While the paper states that methods within the same task use the same input, it fails to clearly explain how the second input for the Fusformer was obtained, whether the original model was modified, or whether different methods obtained equal amounts of input information. These details affect the fairness of baseline comparisons and the reproducibility of the experiments. It is recommended that the input, adaptation method, and training configuration of each model be clearly provided in the main text or appendix.
- The analysis of ranking changes can be further quantified.
The paper demonstrates the changes in leading methods under different conditions through specific experimental results, thus providing numerical evidence for the observation that "rankings change." However, the existing analysis does not systematically report the magnitude and frequency of ranking changes, and lacks repeated experiments or uncertainty estimates, making it difficult to determine whether some ranking changes with small differences are stable. Furthermore, cross-regional results are mainly summarized by geographical group, and it is not yet possible to determine which countries or land cover types primarily drive the changes. It is recommended to supplement the analysis with cross-conditional ranking consistency indicators, multi-random seed results, and analyses categorized by country or land cover type, and to indicate which methods remain relatively stable under multiple conditions.
Questions for authors
See weakness