- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper presents GenomeCoder, a method for trying to score/identify likely latent variables/explanations () when given observations () for various genetics/genomics-related processes. For example, trying to find the likely exons which are responsible for an observed transcript. The method assumes there is already a mechanistic model which can score , and it attempts to then find , where is the normalization constant. The method does this by training a neural network to propose/prioritize a small set of s to score, and then applies MAP to find the best . The neural network is trained to take in observations and produce likely explanations , via EM to maximize the final . This allows giving an estimate of what the missing probablistic support of is.
Critical strengths
Neural network for prioritization is a nice separation
The neural network only decides what candidate s get scored, and the downstream correction for scoring a limited set of s is a clever workaround for intractability.
Upfront reporting of experimental results
The results are reported in an honest and clear way. The scope of the paper is also stated, in that evaluation is limited by the mechanistic model which also defines the posterior.
Critical weaknesses
Methodology seems somewhat circular
The method being presented relies heavily on the (known) mechanistic model, , both as the source of training data, and also as the evaluator. That is, consists of simulated data which is used to train the neural network which proposes s given . Then, it is used again to provide the exact scores used for inference, and also defines the Bayes ceiling. It is also used to generate test data. So of course, it makes sense that in this setup, GenomeCoder performs well because it's trained specifically to recapitulate the data exactly. Thus, the reported advantage over baselines (e.g. Kallisto/Salmon) is only in the context of being in-distribution for this specific simulated dataset. Its advantage is likely to shrink once we move to real data.
Benchmarks are limited to simple simulations
Almost all accuracy results in the paper are measured against simple simulations (which the method was trained to match). These simulations are generally quite simple (e.g. at most 3 - 5 isoforms per locus for transcript inference). Furthermore, the setup of the method relies heavily on having a well-defined mechanistic model (as described above). So this method has not been demonstrated on real-world data.
Paper writing is unclear
Much of the main text depends on definitions and details that are hidden in the appendices or are missing. It is difficult to decipher what the methodology is, what each piece is required for (i.e. motivation for each part of the methodology). The paper would certainly benefit from a more structured walkthrough of what the proposed method is. Many of the steps are not clearly defined, and their role in the method is also not clear. Some examples of this are listed in the questions below.
Questions for authors
- What is x (the context)? Is it ever used in practice (if so, what form does it take on?) or is it just a mathematical concept used for the theorems?
- What is the goal of the EM algorithm? It seems like it's used to learn but isn't this given directly from the biological model (i.e. the dataset)?