- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces PeakCloud, a benchmark for sample-level disease classification from untargeted MS1 mass spectra, and PeakCloudNet, a set-based neural network for learning directly from peak sets.
The benchmark unifies four cohorts in a shared peak-level format under fixed, subject-disjoint splits: three public LCMS cohorts (MTBLS17 serum HCC vs. cirrhosis, n=262; MTBLS28 urine lung cancer vs. control, n=1004; ST001000 feces IBD vs. control, n=220) and a new breath PTR-TOF-MS cohort (MSP001, n=700 TB vs. healthy controls) in which every subject was measured concurrently on two instruments, with longitudinal acquisition batches and paired ambient-air references used to remove environmental confounds. Evaluation covers in-distribution performance plus three OOD regimes: cross-instrument transfer (both directions, on 140 paired held-out subjects), temporal batch shift, and external-cohort transfer.
PeakCloudNet encodes each peak with a resolution-aware damped-Fourier m/z embedding and a centered log-ratio intensity embedding, fuses them per peak, and pools with a Set Transformer. Peak-CAM attribution plus a frozen pretrained molecular-formula retriever (Appendix A.9) are used for explainability, with candidate formulas matched against previously reported disease-associated metabolites (e.g., 3-heptanone and methylnaphthalenes for TB breath; creatine riboside / NANA / bile acids on the LCMS cohorts).
Validation: 8 baselines (RF, SVM, PLS-DA, PCA+LR, MLP, DeepMSProfiler, LCMS-Net) under identical splits; AUC/BA with 2000-resample bootstrap CIs over 3 seeds. Headline results: PeakCloudNet is best in-distribution on both breath instruments (0.97 AUC) and transfers best across instruments in both directions (losing only 0.02-0.09 AUC, vs. up to 0.21 for RF and a collapse to chance for DeepMSProfiler in one direction); on batch/cohort shifts it is competitive with RF; ablations attribute most of the gain to the m/z encoder.
Critical strengths
- The paired cross-instrument evaluation protocol is a real methodological contribution. Measuring the same 140 subjects on two instruments isolates hardware drift from cohort composition — something no public benchmark allows (the closest precedent, Metabo-Ring, is a QC study without ML evaluation). This design is what makes the paper's central finding — that ID accuracy does not predict cross-instrument robustness, and that a grid-based CNN can collapse to chance (0.50 AUC) in one transfer direction — credible and actionable for multi-site deployment. It is the main reason this paper matters beyond another benchmark table.
- A novel, well-motivated architecture with honest reporting of its own weaknesses. Set-based learning for whole untargeted MS1 profiles (vs. spectrum-level or feature-detection uses of point-cloud methods in prior work) is, to my knowledge, new. tThe resolution-damped Fourier m/z encoding and CLR intensity encoding are physically motivated rather than ad hoc. The authors also report awkward results candidly (RF competitive on batch/cohort shifts; the ablation inversion), which increases confidence in the rest.
Critical weaknesses
- Nothing is accessible at review time, and one key component has no identifiable provenance. The paper contains zero URLs: no code link, no data link, no anonymous repository. For a benchmark paper — whose value is reproducibility and reuse by others — all artifacts are promised "upon publication," with MSP001 additionally contingent on institutional authorization. This also makes the headline dataset claim ("first public breath PTR-TOF-MS cohort with paired cross-instrument measurements") currently unverifiable. The molecular-formula retriever used for the formula-retrieval explainability results is described only as "a pretrained spectral matching model" from an unnamed, uncited "release," with results attributed to a checkpoint whose identity and evaluation sample count are unspecified. A reviewer cannot verify any part of this component, yet it underwrites the abstract's claim that the method "retrieves candidate formulas aligned with known metabolic pathways."
- The evidence does not statistically establish several headline claims, and the abstract overstates them. Multiple key comparisons have overlapping 95% bootstrap CIs with no significance tests or seed-level analysis (MSP001-1 ID vs RF; both cross-batch comparisons; ST001000, where PeakCloudNet is actually below SVM and RF). On MTBLS28, LCMS-Net (0.80 +/- 0.06) beats PeakCloudNet (0.75 +/- 0.06) with non-overlapping CIs, so the abstract's "competitive performance against grid-based models" is generous. The abstract also cites "nine classification tasks," which is never defined or enumerated in the paper, and there is no limitations section discussing that all biological interpretations are putative mass matches with no wet-lab or prospective validation.
Questions for authors
- What is the molecular-formula retriever in Appendix A.9 — its name, citation or link, and training provenance? Can you report its top-1 / top-5 accuracy measured on the attribution-selected peaks actually used in Section 5.4, rather than the "single-peak reference" result from the release?
- How do you count "nine classification tasks" (abstract)? Please enumerate them, since Table 1 lists four datasets with ID/OOD settings.
- For the comparisons where bootstrap CIs overlap (MSP001-1 ID vs RF; cross-batch on both instruments; ST001000), can you provide seed-level results or a significance test? Is the cross-instrument advantage of PeakCloudNet over RF statistically distinguishable in either direction?
- Why does removing both encoders (0.65) hurt more than removing only the m/z encoder (0.72) on MSP001-2, while the opposite ordering holds on MSP001-1? What does this imply about which component drives cross-instrument robustness?
- Will an anonymous code and data link (including processed peak-level MSP001) be available during the review period, or only after acceptance? If release is contingent on institutional authorization, what is the expected timeline? In my opinion, full evidence of reproducibility at the review time is an absolute must for venues like ICLR.
- Are you planning to add wet-lab or targeted confirmation of the top attributed peaks (e.g., 3-heptanone, methylnaphthalenes)? That would convert the explainability story from putative mass matching to validated evidence.