- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes MuRF-FSL, a method for finding shifted features across multiple datasets without needing a clean reference dataset. The main idea is that if most data sources agree on a feature, their shared pattern can be used as a reference to identify which sources have shifted.
The authors use this idea in a neural network and test it on 11 datasets with different types of simulated shifts. The method achieves good results compared to the baselines, including methods that have access to a clean reference.
Critical strengths
-
I like the main idea of the paper. In many real applications, we do not have a clean reference dataset. Using information from multiple sources to build an internal reference is a useful way to address this problem. The method is also flexible and can work with different numbers of sources and features.
-
The experiments are extensive. The authors test the method on 11 datasets with different sizes and feature types. The results are strong, and the ablation study clearly shows that the source-relative component is important. I also appreciate that the method works without retraining on the test datasets.
Critical weaknesses
-
My main concern is that all the shifts used for evaluation are artificially generated. Although the original datasets are real, the feature shifts are introduced through predefined operations. It is not clear how well the method would work on naturally occurring shifts, such as differences between hospitals, scanners, or data collection systems.
-
The method assumes that most sources are normal for each feature. This is an important assumption, but it may not always hold in real applications. For example, several hospitals may use the same processing pipeline and introduce the same error. The experiments always follow this assumption, so we do not know how the method behaves when it is violated.
Questions for authors
-
Have the authors tested the method on any dataset with naturally occurring feature shifts rather than simulated ones? I think such an experiment would help show how useful the method is in real applications.
-
What happens when half or more of the sources have a shift in the same feature? Can the authors show how the performance changes as the number of shifted sources increases beyond the assumed limit?
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with about one day remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.