- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes TaxoSet, an algorithm to retrieve taxa from DNA barcodes when only a handful of taxa are sequences. It proposes a neural network system to decide when the given DNA barcode corresponds to a mapped taxon, whether it corresponds to one that is not included in the references or if it is only related (children) to one of them. In the latter case the algorithm returns a set of all plausible children that could correspond to the given barcode (when no information allows to differentiate between them). The algorithm reaches fairly good classifications capabilities on various markers.
Critical strengths
- the paper addresses an interesting topic which could be particularly useful in practice in biology
- it explains in depth through various experiments and ablations how the method works, its strengths and its weaknesses
Critical weaknesses
- the paper does not clearly describe the training process (no optimization curve, how hyperparameters were chosen, ...). This does not give a clear view of how the core of the algorithm is trained, what are the challenges of training and whether the results found in the paper could actually be improved even further with training adjustments. These can usually be crucial information for future replications and if possible future iterations on this algorithm.
Questions for authors
- The authors discussed briefly the problematic of having invalid queries which could fool the model. What about incoherence in R or T like having the wrong DNA for a given taxa or a mislabelling of branches in T? Would the algorithm be very much affected or can it still retrieve coherent results using the rest of the data? This could be particularly interesting if in the future AI is used to aggressively map the genome of species and could possibly make some mistakes here and there.
- The algorithm was trained for 40 epochs without data augmentation, yet it does not show the training curves. In classical DNN this is likely to lead to severe overfitting - did the authors verified what is going on there and whether the results could be improved by adding regularizers in the loss / data augmentation?
2 comments
Answer to rebuttal
I thank the authors for their additional experiments. I raised my confidence.
We thank the reviewer for the questions. Both led to new experiments.
Training process. Appendix B.2 now reports the hyperparameters and where they come from. They were carried over unchanged from an earlier campaign and are the same for all four markers. We ran no search and do not claim that they are optimal.
Q2, overfitting over 40 epochs. Figure B.2 now shows the training curves of the four markers. The held-out loss reaches its minimum between epochs 14 and 34 and then stays within 0.01 to 0.07 of it until epoch 40: we see no overfitting. We also tested the one augmentation that can act on the network's inputs: masking 10% or 20% of the retrieved sequences during training moves no metric by more than 0.5 points (Table B.1; ITS, one seed).
Q1, errors in the reference and the hierarchy. Appendix H now tests both on ITS, with one seed. With 5% of reference sequences relabelled to another genus, the network loses 0.5 points of genus accuracy and 1.1 of species accuracy on known-species queries, versus 4.6 and 3.5 for best-hit on the same retrieved sequences: reading 20 sequences absorbs a wrong label that the closest sequence alone passes on. With 5% of the hierarchy's branches moved to another parent, routing rate and coverage change by less than one point (Table H.1). This noise is uniform and synthetic; real errors cluster, and Appendix H and Section 6 keep that caveat.