- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes whether a claimed advantage of a model, as shown by metrics, is trustworthy. It identifies some serious potential issues, including variability in random seeds, changing evaluation criteria, and missing baselines. It then proposes a way to preregister an experiment tied to a fixed configuration, evaluation metric, and random seed selection. It then provides a case study of developing models in spatial-temporal forecasting to highlight these potential issues and how the method can partially fix the problems.
Critical strengths
- The paper correctly identifies the potential issues in common practices in the current model and method search. These issues are important to avoid and often lead to false claims about the effectiveness of a proposed method.
Critical weaknesses
- Preregistering an experiment does not eliminate selection bias. The criterion that the improvement must exceed one standard deviation also seems a bit arbitrary, and it is unclear whether it is too restrictive or too lenient, or if it should depend on the specific domain and problem. In addition, one can still achieve apparent improvements by chance simply by trying many different methods and selecting the best-performing one, so the method does not seem to avoid selection bias.
- The paper only provides a case study in the spatio-temporal forecasting domain. The discussion of the problem and the proposed solution should not be tied to this specific domain, and the empirical study will be much stronger if extended to more applications.
Questions for authors
- The issue studied in the paper is a fundamental one, and should not be limited to just the spatial-temporal domain. Similar biases happen in virtually any other domain that involves neural network training. Is there a particular reason for focusing on spatio-temporal forecasting, and is there anything specific about the proposed method that only applies to this domain?
- The proposed protocol evaluates each experiment independently, but one could technically still conduct a large number of preregistered experiments with different modifications and obtain apparent improvements simply by chance. For instance, under some normal hypothesis, using the 1 standard deviation test, the proposed method will incorrectly accept roughly 16% of false positive improvements. In light of this, could the framework also incorporate quantities such as the total number of experiments conducted (e.g., when searching through a hyperparameter space) when determining whether an improvement is statistically significant, thereby accounting for this form of selection bias?