- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The manuscript introduces an active-planning workflow, ActiveMS, for prioritizing candidates from MS2 spectrum prediction for manual verification. A benchmark for active search methods on this task is presented. Two methods for predicting spectra are evaluated in combination with ActiveMS, both resulting in improvements with regard to the necessary rounds of verification.
Critical strengths
- The application of the active planning idea to spectrum prediction candidates is carried out thoroughly, including three reasonable baselines (besides random and normal ranking also Feedback Greedy), and ablation studies. The benchmark is split into two parts, one based on commercial NIST libraries, the other on a recent, publicly available, and curated spectral library.
Critical weaknesses
- While improvements on the proposed benchmarks are sound, the practical usefulness of the method is very questionable, hence our recommendation for rejection: It is assumed that we know at which point we have reached identification, which is not the case in practice. Thresholds on spectral similarity or the minimum number of matching peaks are not discussed. Furthermore, the term "identification" should rather be avoided: for the highest level of annotation confidence, we need further orthogonal proof (https://doi.org/10.1021/es5002105); highly similar compounds may result in indistinguishable spectra. The feedback part of the benchmark is based solely on simulated spectra, carrying the risk that the proposed method might not work with "real" reference spectra. The multi-layer neural network architecture of ActiveMS, including cross-attention, could easily lead to overfitting on this particular feedback from simulated spectra. With a large number of high-quality reference spectra publicly available, a smaller complementary benchmark may show that the methods can indeed work also for feedback from non-simulated spectra.
- Many decisions in the methodology lack motivation or seem rather arbitrary: for instance, three different similarity measures between spectra are used at different points: spectral entropy, cosine, and Euclidean distance, with no explanation. The reason why we need a prioritization method for candidates is not clearly stated, or backed up by references (for example the assumed cost of manual verification, which can only be found in Figure 1, nowhere in the text). The structure-to-spectrum paradigm that spectrum prediction methods are a part of is only one of two main approaches for compound annotation from tandem mass spectra, the other being the spectrum-to-structure paradigm, which is not mentioned or discussed anywhere. The manuscript reads more like another instance of the active planning idea being applied to a new problem. We do not find substantial methodological novelty compared to existing active planning methods.
Minor:
- y-axis scaling in Figure 3 is very strange and not explained
- Recheck bibliography: e.g., link to Garnett 2012 paper wrong, same for Jiang 2018
- "Not Use Feedback" in Figure 1
Questions for authors
- How does the identification success from the benchmark translate to real-world use cases? What are the criteria to determine success?
- What are the reasons for the different similarity/distance measures for mass spectra applied throughout the manuscript?
- What are the methodological novelties of the employed active planning idea compared to existing methods/frameworks?