- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The manuscript introduces the method GEMS (Guided Editing of Molecular Structures) for the structure elucidation of small molecules based on tandem mass spectrometry data. Given a molecular formula, GEMS uses MIST (Goldman et al., 2023) to predict the molecular fingerprint of the unknown molecule based on its spectrum. It then selects the candidate structures matching the best to the predicted fingerprint. As the true structure doesn't have to be among all candidates and as those selected candidates might not fit entirely well to the predicted fingerprint, GEMS employs the idea to apply edit operations to these selected candidates to propose new structures that fit better to be predicted fingerprint. These edit operations are chosen with a neural network that scores all those valid edit operations based on the predicted fingerprint and current candidate structure.
Critical strengths
- Although the refinement of candidate structures is not entirely novel, GEMS' approach of using the predicted fingerprint of the unknown molecule to direct the editing of the candidate structures is new. For comparison, FOAM (Manjrekar et al., 2026) applies random mutations to generate new candidates and FRIGID's (Bohde et al., 2026) generation of new candidates is guided by predicted spectrum consistency. As the predicted fingerprint already contains information about the possible presence or absence of structural features, GEMS' approach is totally reasonable.
- Furthermore, the advantage of GEMS' approach is that the editing is only guided by predicted structural features. This enables that the navigation policy for editing can solely be trained on structure databases, which are orders of magnitude larger than spectral libraries.
Critical weaknesses
- The evaluation of GEMS against purely de novo methods is somewhat unfair, as GEMS generates new structures from already proposed structure candidates (de novo generated and/or database retrieved molecules). Therefore, a fairer evaluation would be to generate as many candidates per de novo method as GEMS incorporates into its total candidate pool.
- GEMS was also evaluated when discarding PubChem candidates with a Tanimoto similarity greater than 0.7 to the true molecule. Using Tanimoto similarity seems questionable, as GEMS uses edit operations to explore new candidates and as the MCES distance is a direct measure of these edit operations. Therefore, it would be more reasonable to perform an additional evaluation where all candidates with an MCES smaller than 2, 4 or 6 to the true molecule were removed.
Minor:
- The manuscript sometimes chooses overly confusing phrasings, for instance:"retain the highest-scoring representative of each InChIKey14 group" instead of "We rank PubChem molecules with formula F by s(m;q), not considering stereochemistry, and select up to K inital structures"
- Expressions like "clip" should be mathematically defined
Questions for authors
- The navigation policy of selecting the best edit operations is pre-trained on PubChem isomer pairs. How do you ensure that the benchmarking does not encounter any of the training pairs?
- How does the method perform without fine-tuning (ablation study)?
- The method should naturally perform best, if the true molecule is part of the initial candidate set. Could you evaluate this to make sure that the navigation policy does not prefer unnecessary edits?
3 comments
Reply to Reviewer RZko, part 2 of 2
Q1. Can the benchmark contain training pairs?
No, this cannot happen. We remove the held-out structures before we build any training pair. This is described in Appendix B.1 and in footnote 1 of Section 4.4.
Before we mine the pretraining pairs, we remove every validation structure and every test structure of both benchmarks from the PubChem sources and targets. We identify structures by their connectivity (InChIKey14). We also discard every intermediate structure on a path if it has the connectivity of a validation or test structure. The fine-tuning data uses the same exclusion. As a result, no training pair contains a test structure as its source, as its target or as an intermediate structure.
For this rebuttal, we also made a direct check. The policies that are trained from scratch use 216,221 distinct structures on NPLIB1 and 845,538 on MassSpecGym. None of them is a validation structure or a test structure. We will move the description of this exclusion from the appendix to the main text.
Q2. The performance without fine-tuning
The NPLIB1 result without fine-tuning is already in Table 5. We now ran the same experiment on MassSpecGym. It uses the same inputs, the same search and the same ranking as the paper. The table below shows the results in percent.
| Policy | NPLIB1 Hit@1 | NPLIB1 Hit@10 | MassSpecGym Hit@1 | MassSpecGym Hit@10 |
|---|---|---|---|---|
| Pretrained only, without fine-tuning | 28.89 | 46.82 | 27.84 (new) | 36.04 (new) |
| Pretrained and fine-tuned (paper) | 29.64 | 50.81 | 30.22 | 41.33 |
| Strongest baseline in Table 1 | 25.03 | 33.37 | 23.23 | 27.70 |
The strongest baseline is FRIGID on NPLIB1 and our MS-GPT reproduction on MassSpecGym.
The pretrained policy is trained only on pairs of PubChem isomers with exact fingerprints. It has never seen a spectrum or a fingerprint that was predicted by MIST. It is already better than every baseline in Table 1 on both datasets.
Fine-tuning increases Hit@1 by 0.8 points and Hit@10 by 4.0 points on NPLIB1. On MassSpecGym, it increases Hit@1 by 2.4 points and Hit@10 by 5.3 points. The main reason is that the fine-tuned policy reaches more targets. We will add the new MassSpecGym result to Table 5.
Q3. The target among the starting candidates
We ran the test searches again with the models and the settings of the paper. We changed only one thing. The target is no longer removed from the five PubChem starts. The table below shows the results in percent.
| Setting | NPLIB1 Hit@1 | NPLIB1 Hit@10 | NPLIB1 reach | MassSpecGym Hit@1 | MassSpecGym Hit@10 | MassSpecGym reach |
|---|---|---|---|---|---|---|
| Target excluded (paper) | 29.64 | 50.81 | 57.78 | 30.22 | 41.33 | 46.88 |
| Target allowed | 34.37 | 61.39 | 73.23 | 35.04 | 52.73 | 66.52 |
As you expected, GEMS performs best when the target can be among the starts.
We also looked at whether the search makes unnecessary edits. The search never removes a starting candidate. Therefore, a correct start always stays in the final candidate pool.
We then looked at the queries where the top-ranked start is the target. In 79% of these queries on NPLIB1 and in 75% on MassSpecGym, the target stays at rank 1 after the search. In most of the other queries, the search creates a structure that is not in PubChem, and this structure fits the fingerprint predicted by MIST better than the true structure does. So this effect comes from the final ranking with imperfect fingerprint predictions, which we discuss in Section 5.6. It does not come from the navigation policy. The consensus term is also not the reason. When we rank only by the fingerprint score, the target loses rank 1 in even more queries.
Our main experiments use the harder setting, where the target is not in the database and every correct answer needs at least one edit. We will add this table to the appendix.
Minor points
Thank you for these suggestions. We will define the clip function where it first appears in Equation 2, as clip(x, a, b) = min(max(x, a), b). We will also use your sentence for the retrieval step. The new text will be "We rank same-formula PubChem molecules by s(m; q), ignoring stereochemistry, and select up to K = 5 starting structures." We will also look for similar sentences in the rest of the paper and make them simpler.
Thank you, all our problems have been addressed. We changed our score to 4 (clear accept).
Reply to Reviewer RZko, part 1 of 2
Thank you for your detailed review. We ran the evaluation with MCES filtering that you suggested. We also ran new experiments for Q2 and Q3.
W1. The comparison with de novo methods
FRIGID already uses a larger candidate budget than GEMS. It tries to generate 128 samples per spectrum in each of its 25 rounds. This gives 3,200 samples per spectrum. GEMS constructs about 1,000 structures per query. On average, it ranks 163 of them on NPLIB1 and 177 of them on MassSpecGym.
MS-GPT generates 100 samples per spectrum. GEMS also works well with a similar budget. With a search width of 4 instead of 16, GEMS constructs at most 180 structures per query and ranks at most 65 of them. It still reaches 27.65% Hit@1 and 47.07% Hit@10 on NPLIB1, and 29.18% and 39.39% on MassSpecGym (Table 6). These values are higher than the values of every baseline in Table 1.
GEMS also needs few model evaluations to produce its candidates. It evaluates its policy at most 165 times per query, and at most 45 times with a search width of 4. MS-GPT generates each of its 100 samples token by token. An NPLIB1 test molecule has about 59 tokens on average, so this takes several thousand decoder steps per spectrum. FRIGID uses several denoising steps for each of its 3,200 samples, and it also runs ICEBERG on its candidates.
The GEMS policy is also much smaller. It has 12.6 million parameters. The generators of MS-GPT and FRIGID have 150.1 million and 182.2 million parameters.
Table 1 also contains results where GEMS does not use a database. In these experiments, GEMS starts from the samples of a generator. On NPLIB1, we start GEMS from the initial samples of FRIGID-base. From this pool, GEMS reaches 28.02% Hit@1 and 43.34% Hit@10. FRIGID refines the same pool with 24 rounds of ICEBERG. In our run, FRIGID reaches 24.03% and 33.13%. The published FRIGID values are 25.03% and 33.37%.
In the revision, we will divide Table 1 into two groups. One group will contain the methods that use a database at inference. The other group will contain the methods that do not use a database.
W2. Removing close starts with the MCES distance
We ran this experiment for all three thresholds. For each query, we go through the PubChem isomers in the order of their MIST ranking. We keep the first five isomers that have an MCES distance of at least the threshold to the target. We use the same MCES implementation as for the MCES@k values in the paper. All other settings are the same as in the paper.
The table below shows Hit@1 and Hit@10 in percent. The last two rows show the two strongest baselines for comparison.
With a threshold of 2, the results almost do not change. With a threshold of 4, GEMS is still better than every baseline in Table 1 in Hit@1 and Hit@10 on both datasets.
A threshold of 6 removes most of the useful starts. After this filter, the median start has an MCES distance of 10 to the target on NPLIB1 and of 20 on MassSpecGym. For 32% of the NPLIB1 queries and 59% of the MassSpecGym queries, even the closest start has a distance of at least 10. One edit changes at most four bond-order units, so it usually changes the MCES distance by 4 or less. Therefore, a start at a distance of 10 usually needs at least three edits, and a start at a distance of 20 usually needs at least five edits.
In this case, GEMS can search deeper. The policy does not use the search depth as an input. Therefore, we can use the same model for a deeper search without new training. With 20 edit rounds, GEMS is better than FRIGID on both datasets, and it is better than every baseline on NPLIB1. On MassSpecGym, its Hit@1 is below our MS-GPT reproduction, and its Hit@10 of 27.65% is almost equal to the 27.70% of this reproduction.
MS-GPT does not use the database, so this filter does not affect it. When PubChem has no close candidate, GEMS can start from the outputs of our MS-GPT reproduction instead. From these starts, GEMS reaches 25.56% Hit@1 and 34.07% Hit@10 on MassSpecGym (Table 1). The reproduction alone reaches 23.23% and 27.70%.
The deeper search is still fast. With 20 edit rounds, the search takes about 1.6 seconds per query on NPLIB1 and about 2.2 seconds per query on MassSpecGym, on one H100 GPU.
We will add these results to Appendix H.2, next to the results with Tanimoto filtering.