- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a hyperbolic flow-based method for remote sensing change detection. While this approach offered a novel perspective on the problem and demonstrated strong performance on hierarchical datasets, it underperformed on conventional datasets and failed to outperform existing CISS methods. Given that most publicly available remote sensing change detection datasets lack a hierarchical structure, further discussion is needed regarding how to address this limitation and improve performance in such contexts.
Critical strengths
- The authors successfully applied the CISS technique to change detection tasks using remote sensing imagery.
- Furthermore, they demonstrated the potential for incremental change detection by representing semantic information in hyperbolic space and utilizing its flow, achieving strong performance on two prominent public datasets.
- To validate this approach, they introduced a dataset named "HiChange" and conducted a validation study.
Critical weaknesses
-
Table 1 suggests that the proposed method achieves overwhelming performance. However, while the difference in accuracy compared to AnyChange or SAM3 is merely 0.2 to 0.3, the figures for F1-score, Precision, and Recall show disparities of several-fold magnitude. The numerical values presented in this table warrant a thorough verification.
-
Although the proposed method was well-suited to the HiChange dataset, which features a hierarchical structure, most existing datasets lack such a structure. Consequently, the method performs poorly on the second dataset listed in Table 1.
-
Table 3 compares the proposed method with other CISS-based approaches; despite being tailored specifically for remote sensing, the proposed method shows performance that is either similar to or lower than that of existing methods. Additionally, the specific significance of the hyperbolic representation used here needs to be clarified.
Questions for authors
-
There are many change detection datasets (whether binary or non-binary) that do not have a hierarchical structure; would the validity of the proposed method be verified using these datasets?
-
As noted in the section regarding weaknesses, the figures in Table 1 appear questionable; I would appreciate it if you could verify them again.
-
Regarding Table 3, the proposed method does not appear to outperform existing CISS methods. A detailed analysis and explanation of the proposed method's advantages over these existing approaches are required.
3 comments
Thank you for author's response. I have read the author's response to my questions, but it has not changed my evaluation. Then I have decided to maintain my score.
We acknowledge the reviewer's decision of keeping the score. Out of respect for the time and investment we put in addressing all reviewer's concern, we believe however that the reviewer should provide justifications for their choice. We appeal to the principle of "constructive reviews" and ask the reviewer to provide a justification for their decision, in the same way as the reviewer asked us to provide justifications and address their concerns.
A good review should be constructive and help the authors to improve the quality of their paper, even when rejected. It is upon such principles that science is built.
We thank the reviewer for the feedback and for recognizing the potential of applying CISS techniques to incremental change detection, the novelty of hyperbolic flow formulation which achieves strong performance, as well as the value of the proposed HiChange dataset. We address each concern below.
Weakness 1 / Question 2 - Discrepancy between Accuracy and other metrics in Table 1
The values in Table 1 are correct. The observation of small differences in accuracy despite large differences in F1, Precision and Recall is a consequence of the large class imbalance in CD datasets. The SECOND dataset presents only 20% of changed pixels, against 6.6% for Hi-UCD-mini and 5.5% for HiChange. Consequently, accuracy alone is not informative: a trivial classifier that always predicts “unchanged” could already achieve 95% accuracy on HiChange, without detecting any changes. This also explains why small differences in accuracy can be related to large ones in F1, Precision and Recall. We report accuracy only for completeness, as it is a commonly used metric, but since F1, Precision and Recall focus directly on the detection of the minority (=changed) class, they provide a better assessment of CD performance under such class imbalance. In particular, F1-score is the most relevant metric for evaluating CD performance, as it jointly accounts for Precision and Recall, and on this metric HyperFlow clearly outperforms competing methods. To avoid any further confusion, we have removed Accuracy from the reported results.
Weakness 2 / Question 1 - Performance on non-hierarchical datasets
We respectfully disagree. We evaluate HyperFlow on two datasets that do not present such a hierarchical structure (SECOND and Hi-UCD-mini), and on these datasets HyperFlow clearly outperforms the baselines on F1. On SECOND, Hyperflow achieves a F1 of 43.0 against 34.7 for the best unsupervised baseline, and on Hi-UCD-mini 44.3 against 26.5.
Weakness 3 / Question 3 - Comparison with CISS methods in Table 3
The goal of our work is to tackle the problem of class-incremental semantic change detection (CI-SCD), not to outperform dedicated CISS methods on CISS, which is not the task that we are addressing. Table 3 is meant to show that HyperFlow achieves state-of-the-art CISS performance, while additionally being able to perform change detection, which competing CISS methods cannot do. Table 3 shows that this added capacity to perform CD does not come at the cost of incremental segmentation.
Summary
We sincerely appreciate the reviewer's careful assessment. Most of the concerns stem from evaluating the results through accuracy only, a metric that is not suited to the highly imbalanced class distribution of CD. When the results are read through the relevant metrics (F1, precision, and recall), our method outperforms the baselines on both hierarchical and non-hierarchical datasets. In our response, we addressed all comments raised.