- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduce SOUR predicts molecular structures from mass spectra and known molecular formulae. It selects query-relevant synthetic spectra to train a spectrum-to-fingerprint encoder, then further adapts the encoder using query pseudo-labels. A molecular decoder is trained with perturbed fingerprints and learns to revise candidate structures using feedback from simulated spectra. On MassSpecGym and NPLIB1, SOUR achieves strong improvements in fingerprint prediction and retrieval.
Critical strengths
- Comprehensive evaluations on the MassSpecGym and NPLIB1 benchmarks demonstrate the effectiveness of SOUR.
- The approach is general and model-agnostic, with encoder adaptation improving performance across multiple backbones.
Critical weaknesses
- The broad state-of-the-art claim is not fully supported by the reported comparisons. Published FRIGID results on MassSpecGym report 18.29% Top-1 accuracy, compared with 17.98% for SOUR. This raises concerns about the accuracy and fairness of the paper’s claims.
- The models are trained and adapted separately on MassSpecGym and NPLIB1, with no cross-dataset evaluation. In real-world applications, a suitable target dataset may not be available for training or validation. It is therefore unclear whether the approach can generalize to new data sources in such settings, which limits its practical applicability.
Questions for authors
- Why does NPLIB1 use up to 2,048 candidate molecules per query, while MassSpecGym uses only 64? Could you evaluate both datasets with (K = 64, 256, 2048), using the same collision energies and keeping the encoder update budget and real-data mixing fixed within each dataset?
- Could you compare random and selected synthetic data, both with and without Phase 2, using different data sizes, such as 10K, 100K, and 1M synthetic pairs?
- MassSpecGym has more than 10,000 test spectra, yet the adaptation process uses 20 million synthetic pairs. This raises concerns about the computational and storage costs of generating and maintaining such a large synthetic dataset, which may limit practical use.
- Could you report accuracy by the maximum similarity to the decoder pretraining set and the synthetic-source molecules, using bins below 0.5, 0.5-0.7, 0.7-0.9, and at least 0.9? This would help assess how well the method generalizes to molecules that are less similar to the training data.
4 comments
Why does NPLIB1 use up to 2,048 candidate molecules per query, while MassSpecGym uses only 64? Could you evaluate both datasets with (K = 64, 256, 2048), using the same collision energies and keeping the encoder update budget and real-data mixing fixed within each dataset?
This difference in the molecule allowances for Phase 1 can be attributed to the differing query spectrum counts of both benchmarks: NPLIB1 consists of 819 test spectra, while MassSpecGym contains 17,556 test spectra (about 21x more). To achieve a reasonably similar order of total synthetic data points to utilize similar compute for adaptation, we allot a higher number of candidates to be accrued per query spectrum for NPLIB1 versus MassSpecGym. We do not have sufficient compute to consult and adapt on significantly more additional queries for MassSpecGym, since e.g. 2048 candidates per query x 17,556 query spectra x 5 collision energies would amount to 179 million synthetic pairs. Our updated Figure 2, which now breaks down the SOUR-selected augmentation into Phase 1 alone and Phase 1+2, highlights the log-linear trend in Phase 1 scaling that would be commensurate with the chosen value of . The reported SOUR utilized for the MassSpecGym benchmark uses and utilizes 5 million spectra; the lower points leftward are run with smaller values of , which cement the log-linear relationship we observe. We do observe from the rightmost point that utilizes 10 million spectra (which could roughly extrapolate as ) that performance saturates (with encoder accuracy going from 0.507 to .505, and de novo Acc@1 going from 11.711 to 11.626). We have similar trends observed for NPLIB1, where scaling the number of synthetic pairs also shows log-linear improvements on encoder reconstruction accuracy:
| Synthetic pairs | Mean Tanimoto |
|---|---|
| 1,000 | 0.448 |
| 5,000 | 0.456 |
| 10,000 | 0.454 |
| 62,500 | 0.485 |
| 250,000 | 0.520 |
| 1,000,000 | 0.558 |
| 5,000,000 | 0.599 |
| 13,000,000 | 0.606 |
Here, since the reported NPLIB1 encoder utilizes 13 million synthetic pairs corresponding to a , we can estimate as roughly matching 500K pairs and matching 1.6 million; the values of mean Tanimoto at the closest runs, 0.520 for 250K and 0.558 at 1 million (versus 0.606 for the selected NPLIB1 encoder) support the conclusion that increasing can indeed log-linearly improve the encoder.
Could you compare random and selected synthetic data, both with and without Phase 2, using different data sizes, such as 10K, 100K, and 1M synthetic pairs?
Thank you for the suggestion. We present a comparison of these two phases of encoder-based augmentation for the MassSpecGym benchmark in the updated Figure 2 and Figure 5 (in PDF), which now demonstrate two separate trends for the improvement in SOUR-based augmentation (where Phase 1 alone is denoted in light pink and Phase 1 and 2 together are denoted in dark pink). This plot has also been updated to include evaluations of the same models for de novo structural elucidation accuracy, in addition to fingerprint reconstruction accuracy. We provide a tabular documentation of these results below (also Table 8 in the updated PDF):
Table: Numerical values for the MassSpecGym synthetic-data scaling study. All metrics are reported to three decimal places. Random augmentation and SOUR Phase 1 use the Phase 1 synthetic budget N. Series incorporating Phase 2 additionally include 17,556 query–pseudo-label pairs and use N + 17,556.
| Phase 1 pairs N | With Phase 2 N+17,556 | MIST Tanimoto: Random aug. | MIST Tanimoto: Random + SOUR Phase 2 | MIST Tanimoto: SOUR Phase 1 | MIST Tanimoto: SOUR Phase 1+2 | SOUR-base Top-1 accuracy (%): Random aug. | SOUR-base Top-1 accuracy (%): SOUR Phase 1 | SOUR-base Top-1 accuracy (%): SOUR Phase 1+2 |
|---|---|---|---|---|---|---|---|---|
| 1,000 | 18,556 | 0.243 | 0.239 | 0.254 | 0.309 | 0.000 | 0.518 | 3.389 |
| 5,000 | 22,556 | 0.241 | 0.282 | 0.266 | 0.338 | 0.023 | 0.473 | 4.004 |
| 10,000 | 27,556 | 0.246 | 0.292 | 0.282 | 0.360 | 0.023 | 1.435 | 5.263 |
| 62,500 | 80,056 | 0.242 | 0.289 | 0.345 | 0.417 | 0.000 | 3.321 | 6.943 |
| 250,000 | 267,556 | 0.262 | 0.325 | 0.377 | 0.432 | 0.091 | 5.588 | 8.863 |
| 1,000,000 | 1,017,556 | 0.297 | 0.338 | 0.415 | 0.478 | 0.461 | 9.552 | 12.298 |
| 5,000,000 | 5,017,556 | 0.328 | 0.374 | 0.441 | 0.507 | 1.077 | 11.711 | 15.448 |
| 10,000,000 | 10,017,556 | 0.291 | 0.310 | 0.442 | 0.505 | 0.325 | 11.626 | 15.379 |
(to be continued)
Both plots indicate that Phase 1 augmentation by itself improves over random augmentation at all steps, with the most significant improvements being accrued with larger amounts of synthetic datapoints (e.g. at 100K synthetic pairs and beyond), whereas random augmentation does not change performance drastically. In fact, for de novo elucidation accuracy, it stays largely unchanged at close to 0%. Applying Phase 2 to every SOUR Phase 1 model improves its encoder accuracy and de novo Acc@1. We further highlight the smaller, but still observable benefit of the Phase 2 augmentation to randomly augmented encoders in the MIST Tanimoto: Random + SOUR Phase 2 column, where the increases from MIST Tanimoto: Random are marginal in most cases, except for at 1,000 pairs where encoder accuracy is instead worsened (from 0.243 to 0.239). We therefore credit the strength of the SOUR-augmented encoder to the complementary nature of both augmentation phases. The value of Phase 2 does improve, but to a lesser extent, suggesting that Phase 1, spectral-side alignment, is a critical first step to getting Phase 2 to further improve the encoder significantly.
MassSpecGym has more than 10,000 test spectra, yet the adaptation process uses 20 million synthetic pairs. This raises concerns about the computational and storage costs of generating and maintaining such a large synthetic dataset, which may limit practical use.
Thank you for raising this concern. First, the 20 million figure for MassSpecGym was incorrect and has been updated to the correct 5 million figure. Generating and storing these 5 million spectra takes only ~1 NVIDIA RTX PRO 6000 GPU hour and 2 GB, which we consider not burdensome. We provide total bulk and amortized per-query compute costs in Table 12 and Appendix D.5, which in part account for the regeneration of selected synthetic neighbors (Synthetic generation) and Phase 1 and 2 training to update the encoder. We consider the biggest compute and storage bottleneck to be building the ~100M PubChem synthetic library, which takes 19.5 GPU hours and 40 GB when compressed per acquisition condition. We believe these are not extremely demanding storage or compute costs, and we plan to make the precomputed synthetic libraries available upon deanonymization to remove the recomputation burden for users adapting to new query spaces.
Could you report accuracy by the maximum similarity to the decoder pretraining set and the synthetic-source molecules, using bins below 0.5, 0.5-0.7, 0.7-0.9, and at least 0.9? This would help assess how well the method generalizes to molecules that are less similar to the training data.
Thank you for the suggestion. We now include a table that stratifies de novo accuracy (Acc@1 and Acc@10) based on their maximum similarity to the decoder pretraining set and the Phase 1 (P1) synthetic-source molecules (Table 9, Appendix D.1.3).
For MassSpecGym:
| Max similarity bin (num. of molecules: decoder / P1) | Decoder pretraining | P1 synthetic sources | ||
|---|---|---|---|---|
| Acc@1 | Acc@10 | Acc@1 | Acc@10 | |
| <0.5 (57 / 300) | 1.0 | 1.0 | 0.2 | 0.8 |
| 0.5–<0.7 (636 / 968) | 5.7 | 9.2 | 3.9 | 5.8 |
| 0.7–<0.9 (1,587 / 1,420) | 7.0 | 13.8 | 14.4 | 20.7 |
| ≥0.9 (717 / 309) | 31.6 | 40.0 | 36.5 | 50.1 |
For NPLIB1:
| Max similarity bin (molecules: decoder / P1) | Decoder pretraining | P1 synthetic sources | ||
|---|---|---|---|---|
| Acc@1 | Acc@10 | Acc@1 | Acc@10 | |
| <0.5 (8 / 6) | 0.0 | 0.0 | 0.0 | 0.0 |
| 0.5–<0.7 (99 / 82) | 14.3 | 28.6 | 3.4 | 8.0 |
| 0.7–<0.9 (352 / 332) | 21.1 | 33.2 | 19.2 | 29.7 |
| ≥0.9 (255 / 294) | 29.1 | 46.1 | 32.0 | 52.2 |
The decoder trends we quantify here, which are near-0 for <0.5 Tanimoto similarity, track with the generalization relationships observed by other decoders such as FRIGID ([1], Table 8); this is reasonable given that the fixes we made to the decoder are less about improving its generalization across chemical space and more to adapt to the errors of the particular encoder we train. On MassSpecGym, test structures with close synthetic neighbors included that end up having high structural similarity see a greater improvement in accuracy relative to the decoder pretraining similarity (e.g. at 0.7-0.9, for MassSpecGym, Acc@1 for decoder neighbors is 7.0% while for synthetic neighbors is 14.4%), which would suggest the SOUR-based exposure of the encoder to similar structures improves structural generation metrics versus exposure of the structure during decoding pretraining.
(to be continued)
Thank you once again for your constructive and detailed suggestions, which have strengthened both our empirical analysis and the clarity of the manuscript. We hope that clarifying the updated FRIGID benchmark results, adding the zero-label cross-dataset and similarity-stratified evaluations, and detailing the practical compute and storage costs fully resolve your concerns, and we would greatly appreciate it if you would consider raising your score toward acceptance. Please feel free to let us know if you have any follow-up questions.
Citations:
- [1]: Bohde, Liu et al., FRIGID (v3) https://arxiv.org/abs/2604.16648
- [2] Zhao et al., MS-GPT (v2) https://arxiv.org/abs/2607.23607
We thank the reviewer for their thorough and thoughtful assessment of our work. Firstly, regarding the presentation score of the paper, we have carefully revised and elaborated upon the motivation, methodology and results of the paper in the updated version of the submission; we hope these revisions make it easier to understand the workflow and assess its merits. Below, we address your insightful observations and suggestions for revisions:
We appreciate the reviewer’s attention to this discrepancy. We coordinated closely with FRIGID and MS-GPT authors on the benchmarking results, and based on the issues and suggestions we all agreed on, both papers have recently updated their preprints, and the updated results are consistent with the reported version in this paper. [1,2] The issue was that the shared MIST checkpoint both FRIGID and MS-GPT used was trained on structures that closely mimic test structures, and the only filtering rule was to exclude exact matches. They decided this is not a fair setting because testing information was used (though indirectly) during training, and the FRIGID authors rebuilt those baselines with a fairer version of MIST, the MIST 4M aug. version, which is reproducible and does not select synthetic data unfairly. We note that this differs fundamentally from SOUR's query-conditioned selection: SOUR only uses the test spectra, which are available at inference time, and never uses test structures, whereas the data selection behind the previous MIST checkpoint used information from test structures. On this fixed fair benchmark that all authors agreed on, SOUR demonstrates a step forward in performance on structural elucidation. We hope this detailed history of auditing and revision of results by community members reassures the reviewer about the legitimacy of the claims we present in the paper.
We appreciate the reviewer’s consideration for the real-world utility of our approach. We did not perform naive cross-dataset evaluation in the main text because 645 of the 714 unique test molecules in NPLIB1 (90.3%, including 570 in MassSpecGym's training split) appear directly in MassSpecGym, which would introduce severe train-to-test leakages of data. We agree with the reviewer’s assessment that a suitable labeled training or validation dataset may not exist in some real-world settings.
To evaluate zero-target-label generalization from a general reference corpus (MassSpecGym) to a new specialized source (NPLIB1) without leakage, we isolated the 32 NPLIB1 test spectra whose structures are completely absent from all MassSpecGym splits and synthetic pools, evaluating fixed MassSpecGym checkpoints directly with their MassSpecGym validation thresholds (zero NPLIB1 training or validation data):
Real-only
MIST No-aug.transfers poorly across sources (0.260vs.0.335in-domain). In contrast, synthetic-augmented MassSpecGym encoders generalize well with zero target labels: bothMIST 4M-aug.(0.387) and even the unadapted frozenMSGMISTSOUR checkpoint (0.354) outperform the in-domainNPLIB1 No-aug.encoder (0.335) trained directly on 9,181 labeled NPLIB1 spectra (and on the broader 63-spectrum subset disjoint from MassSpecGym and4M-aug., cross-sourceMIST 4M-aug.reaches0.421, matching in-domainNPLIB1 No-aug.at0.421). Notably,MSGMISTSOUR performs slightly worse thanMSG 4M-aug.here because its weights were adapted specifically to MassSpecGym's test queries QMSG. In real deployment on an unlabeled new data source Qnew, SOUR's test-time adaptation requires no target training or validation labels; one only needs the GLACIER simulator and source training set Dreal to adapt directly to the unlabeled target spectra Qnew.(to be continued)