- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors argue that a common problem exists in evaluating perturbation prediction. The DE gene sets are not precisely predicted.
Critical strengths
The results are interesting with comprehensive statistical studies.
Critical weaknesses
I think the proposed questions are very interesting, and the others do introduce a statistical framework to address them; however, I doubt whether it can really affect practical usage, as well as the leakage from using Gene Ontology features.
- The paper is presented in a form more like an empirical study rather than a rigorous research paper. There are many approaches we can use to evaluate the quality of perturbation predictions, and Pearson and MSE are two naïve choices. Several researchers have explored better metrics, but this manuscript does not discuss them.
- The methods selected in this study are outdated; updated perturbation prediction models, such as STATE and X-cell, are not discussed. The reasons could come from the low performance of selected models and thus it is not reliable.
- The analysis figure in Figure 1 is very weird. The curve is hard to read and understand. Here, I cannot find obvious differences across methods, and I cannot identify the best methods directly.
- The datasets are not precisely assigned. The VCC has more information in public training datasets. The authors also need to consider Tahoe and larger datasets.
- MWU test is just one option for selecting DE gene effects, and since the studies are based on non-parametric and unpaired datasets, the p-value could be altered very easily even with a small change; thus, the authors need to use simulation data to justify their selection first.
- Overall I think this paper is more suitable for conference such as AISTAT or Bioinformatics, rather than a ML-centered conference.
Questions for authors
Please see the weaknesses.