- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This article begins with the observation that, in many recent studies on CD, part of the improvement is attributed to the fact that the proposed method combines features from the two images using descriptors (e.g. feature difference of multiplication) instead of letting the network learn these relationships itself. However, it is never clear whether the improvements actually stem from these interactions between the features of the two images alone, or if the operations used during decoding, in particular the type of fusion, also plays a role. The paper therefore studies whether the benefit of using descriptors depends on how feature are updated and fused during decoding. Using a toy model, they test the four scenarios: with and without descriptors and with joint or independent fusion of the features. The finding is that explicit descriptors help much more under joint refinement.
Critical strengths
-
The motivation of trying to untangle performance improvements is interesting and could be valuable
-
The design of the model and experiments, especially the two-factor experiment, as well as the evaluation protocol is clean
-
Limitations are reported clearly
Critical weaknesses
-
The main, and critical weakness is the limited generalization of the central claim. The article is meant as a general study of what the benefits of descriptors in CD model can be attributed to. Yet the comparison is only done on one architecture and one decoder family, and the central experiments, which is the factorial comparison, is only done on two datasets. The conclusion therefore only holds for this specific configuration and the global takeaway of the paper is therefore narrow. As it is, although very interesting, it sounds more like an ablation study of a given model. To make the paper much stronger and more convincing, you would need to have many recent strong CD methods, both in their BASE and JOINT version, and with and without joint training and make conclusion at a more global level.
-
No external comparison. The paper only compares with its own baselines that are ablations or small modifications of the same architecture, never with existing SOTA models. Again, this make the experiments look more like ablation studies of a broader study.
-
The components are not new. The descriptors mentioned have been used since early deep learning CD work, the architecture is not new. The paper shows that the joint fusion, which is the common way to decode the features, so nothing new, is the best practice. Even if this practice had maybe never been tested, this is a very modest contribution.
-
Given the fact that the paper focuses on how features are processed in the decoding stage, the paper would benefit from a stronger related work section on this part.
Things to improve the paper that did not impact the score:
- Overall the paper is pretty difficult to follow. The paper would for example benefit from a much stronger introduction on what descriptors are and on the problematics that the paper is trying to address.
- Some notations are not defined (K_3x3 for instance), and some are inconsistent, for example between section 3 and Figure 1.
- BASE is mentioned early in the paper, yet define late. It is therefore hard to understand what it is when reading the paper.
Overall, the paper has a good research question and experimental design, and it is nice to underline that papers and CD models should investigate the origine of gains attributed to feature interaction, however the overall contribution of the paper is modest, and is closer to a careful ablation study. Only one architecture and two datasets are used for the main experiment that support the claim, yet the claim of the paper is global. The authors need to prove that the conclusions also hold for other architectures and other recent CD models.
Questions for authors
-
Is there an explanation or an intuition behind the fact that jointly refining the features works better than refining them independently?
-
The experiments are made with both descriptors at the same time. If you were to consider them separately and make the experiments one descriptor at a time, would the results be different? Does one descriptor benefits from the joint refinement more than the other?
My core concerns remain for this manuscript, especially regarding the tchnical contribution and the paper's motivation. Other reviews raised similar concerns. I thought the authors need to consider all these perspecitves to prepare a substantially improved version. So I maintain my origianl score.