acceptodds
Under review as a conference paper at ICLR 2027

GEMS: MOLECULAR STRUCTURE ELUCIDATION AS LEARNED SEARCH OVER ISOMER GRAPHS

Abstract

Identifying small-molecule structures from molecular formulas and tandem mass spectra remains a bottleneck in untargeted metabolomics and natural-product discovery. Even when database retrieval or de novo prediction does not identify the exact molecular structure, the resulting candidates can provide starting points for exploring nearby structures through a few bond edits. We introduce GEMS (Guided Editing of Molecular Structures), which refines candidates through learned 2-switch bond edits that preserve the molecular formula and each atom's total valence. Molecular fingerprints encode local structural features and can be computed directly from candidate structures. GEMS uses MIST, a pretrained spectrum-to-fingerprint model, to predict fingerprint probabilities once per spectrum. A neural policy compares the candidate fingerprint with these predictions, links present features to the atoms and bonds that generate them, and scores all valid edits in one forward pass, allowing each evaluation to propose multiple candidate molecules. Shortest-path supervision enables pretraining on molecular structures alone, followed by fine-tuning with spectrum-predicted fingerprints. At inference, GEMS searches from multiple initial candidates matching the supplied molecular formula and ranks candidates by fingerprint agreement and multi-start consensus. With the target structure excluded from PubChem initialization, three edit rounds achieve Hit@1/Hit@10 of 29.64%/50.81% on NPLIB1 and 30.22%/41.33% on MassSpecGym, exceeding published baselines under comparable settings. On NPLIB1, GEMS achieves higher identification accuracy in about 3.3 minutes end to end, compared with 26.2 hours for our FRIGID run. GEMS also improves identification from generated candidates, including those refined by FRIGID. Experiments with exact target fingerprints identify fingerprint prediction error as a bottleneck in target recovery and final ranking. Deeper search recovers additional targets without retraining, with larger gains from weaker initial candidates.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Molecular structure elucidation, tandem mass spectrometry, molecular graph editing, learning to search

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_RZkoOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript introduces the method GEMS (Guided Editing of Molecular Structures) for the structure elucidation of small molecules based on tandem mass spectrometry data. Given a molecular formula, GEMS uses MIST (Goldman et al., 2023) to predict the molecular fingerprint of the unknown molecule based on its spectrum. It then selects the candidate structures matching the best to the predicted fingerprint. As the true structure doesn't have to be among all candidates and as those selected candidates might not fit entirely well to the predicted fingerprint, GEMS employs the idea to apply edit operations to these selected candidates to propose new structures that fit better to be predicted fingerprint. These edit operations are chosen with a neural network that scores all those valid edit operations based on the predicted fingerprint and current candidate structure.

Critical strengths
  • Although the refinement of candidate structures is not entirely novel, GEMS' approach of using the predicted fingerprint of the unknown molecule to direct the editing of the candidate structures is new. For comparison, FOAM (Manjrekar et al., 2026) applies random mutations to generate new candidates and FRIGID's (Bohde et al., 2026) generation of new candidates is guided by predicted spectrum consistency. As the predicted fingerprint already contains information about the possible presence or absence of structural features, GEMS' approach is totally reasonable.
  • Furthermore, the advantage of GEMS' approach is that the editing is only guided by predicted structural features. This enables that the navigation policy for editing can solely be trained on structure databases, which are orders of magnitude larger than spectral libraries.
Critical weaknesses
  • The evaluation of GEMS against purely de novo methods is somewhat unfair, as GEMS generates new structures from already proposed structure candidates (de novo generated and/or database retrieved molecules). Therefore, a fairer evaluation would be to generate as many candidates per de novo method as GEMS incorporates into its total candidate pool.
  • GEMS was also evaluated when discarding PubChem candidates with a Tanimoto similarity greater than 0.7 to the true molecule. Using Tanimoto similarity seems questionable, as GEMS uses edit operations to explore new candidates and as the MCES distance is a direct measure of these edit operations. Therefore, it would be more reasonable to perform an additional evaluation where all candidates with an MCES smaller than 2, 4 or 6 to the true molecule were removed.

Minor:

  • The manuscript sometimes chooses overly confusing phrasings, for instance:"retain the highest-scoring representative of each InChIKey14 group" instead of "We rank PubChem molecules with formula F by s(m;q), not considering stereochemistry, and select up to K inital structures"
  • Expressions like "clip" should be mathematically defined
Questions for authors
  • The navigation policy of selecting the best edit operations is pre-trained on PubChem isomer pairs. How do you ensure that the benchmarking does not encounter any of the training pairs?
  • How does the method perform without fine-tuning (ablation study)?
  • The method should naturally perform best, if the true molecule is part of the initial candidate set. Could you evaluate this to make sure that the navigation policy does not prefer unnecessary edits?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer RZko, part 1 of 2

Thank you for your detailed review. We ran the evaluation with MCES filtering that you suggested. We also ran new experiments for Q2 and Q3.

W1. The comparison with de novo methods

FRIGID already uses a larger candidate budget than GEMS. It tries to generate 128 samples per spectrum in each of its 25 rounds. This gives 3,200 samples per spectrum. GEMS constructs about 1,000 structures per query. On average, it ranks 163 of them on NPLIB1 and 177 of them on MassSpecGym.

MS-GPT generates 100 samples per spectrum. GEMS also works well with a similar budget. With a search width of 4 instead of 16, GEMS constructs at most 180 structures per query and ranks at most 65 of them. It still reaches 27.65% Hit@1 and 47.07% Hit@10 on NPLIB1, and 29.18% and 39.39% on MassSpecGym (Table 6). These values are higher than the values of every baseline in Table 1.

GEMS also needs few model evaluations to produce its candidates. It evaluates its policy at most 165 times per query, and at most 45 times with a search width of 4. MS-GPT generates each of its 100 samples token by token. An NPLIB1 test molecule has about 59 tokens on average, so this takes several thousand decoder steps per spectrum. FRIGID uses several denoising steps for each of its 3,200 samples, and it also runs ICEBERG on its candidates.

The GEMS policy is also much smaller. It has 12.6 million parameters. The generators of MS-GPT and FRIGID have 150.1 million and 182.2 million parameters.

Table 1 also contains results where GEMS does not use a database. In these experiments, GEMS starts from the samples of a generator. On NPLIB1, we start GEMS from the initial samples of FRIGID-base. From this pool, GEMS reaches 28.02% Hit@1 and 43.34% Hit@10. FRIGID refines the same pool with 24 rounds of ICEBERG. In our run, FRIGID reaches 24.03% and 33.13%. The published FRIGID values are 25.03% and 33.37%.

In the revision, we will divide Table 1 into two groups. One group will contain the methods that use a database at inference. The other group will contain the methods that do not use a database.

W2. Removing close starts with the MCES distance

We ran this experiment for all three thresholds. For each query, we go through the PubChem isomers in the order of their MIST ranking. We keep the first five isomers that have an MCES distance of at least the threshold to the target. We use the same MCES implementation as for the MCES@k values in the paper. All other settings are the same as in the paper.

The table below shows Hit@1 and Hit@10 in percent. The last two rows show the two strongest baselines for comparison.

Starting candidatesSearch depthNPLIB1 Hit@1NPLIB1 Hit@10MassSpecGym Hit@1MassSpecGym Hit@10
Target excluded (paper)329.6450.8130.2241.33
MCES at least 2329.6450.8130.1841.34
MCES at least 4326.5343.3424.0631.91
MCES at least 6321.6733.8714.6719.23
MCES at least 61025.6540.4718.7125.44
MCES at least 62027.5243.2120.5627.65
FRIGID25.0333.3718.2922.00
MS-GPT (our reproduction)21.0528.6423.2327.70

With a threshold of 2, the results almost do not change. With a threshold of 4, GEMS is still better than every baseline in Table 1 in Hit@1 and Hit@10 on both datasets.

A threshold of 6 removes most of the useful starts. After this filter, the median start has an MCES distance of 10 to the target on NPLIB1 and of 20 on MassSpecGym. For 32% of the NPLIB1 queries and 59% of the MassSpecGym queries, even the closest start has a distance of at least 10. One edit changes at most four bond-order units, so it usually changes the MCES distance by 4 or less. Therefore, a start at a distance of 10 usually needs at least three edits, and a start at a distance of 20 usually needs at least five edits.

In this case, GEMS can search deeper. The policy does not use the search depth as an input. Therefore, we can use the same model for a deeper search without new training. With 20 edit rounds, GEMS is better than FRIGID on both datasets, and it is better than every baseline on NPLIB1. On MassSpecGym, its Hit@1 is below our MS-GPT reproduction, and its Hit@10 of 27.65% is almost equal to the 27.70% of this reproduction.

MS-GPT does not use the database, so this filter does not affect it. When PubChem has no close candidate, GEMS can start from the outputs of our MS-GPT reproduction instead. From these starts, GEMS reaches 25.56% Hit@1 and 34.07% Hit@10 on MassSpecGym (Table 1). The reproduction alone reaches 23.23% and 27.70%.

The deeper search is still fast. With 20 edit rounds, the search takes about 1.6 seconds per query on NPLIB1 and about 2.2 seconds per query on MassSpecGym, on one H100 GPU.

We will add these results to Appendix H.2, next to the results with Tanimoto filtering.

AuthorsOpenReview ↗

Reply to Reviewer RZko, part 2 of 2

Q1. Can the benchmark contain training pairs?

No, this cannot happen. We remove the held-out structures before we build any training pair. This is described in Appendix B.1 and in footnote 1 of Section 4.4.

Before we mine the pretraining pairs, we remove every validation structure and every test structure of both benchmarks from the PubChem sources and targets. We identify structures by their connectivity (InChIKey14). We also discard every intermediate structure on a path if it has the connectivity of a validation or test structure. The fine-tuning data uses the same exclusion. As a result, no training pair contains a test structure as its source, as its target or as an intermediate structure.

For this rebuttal, we also made a direct check. The policies that are trained from scratch use 216,221 distinct structures on NPLIB1 and 845,538 on MassSpecGym. None of them is a validation structure or a test structure. We will move the description of this exclusion from the appendix to the main text.

Q2. The performance without fine-tuning

The NPLIB1 result without fine-tuning is already in Table 5. We now ran the same experiment on MassSpecGym. It uses the same inputs, the same search and the same ranking as the paper. The table below shows the results in percent.

PolicyNPLIB1 Hit@1NPLIB1 Hit@10MassSpecGym Hit@1MassSpecGym Hit@10
Pretrained only, without fine-tuning28.8946.8227.84 (new)36.04 (new)
Pretrained and fine-tuned (paper)29.6450.8130.2241.33
Strongest baseline in Table 125.0333.3723.2327.70

The strongest baseline is FRIGID on NPLIB1 and our MS-GPT reproduction on MassSpecGym.

The pretrained policy is trained only on pairs of PubChem isomers with exact fingerprints. It has never seen a spectrum or a fingerprint that was predicted by MIST. It is already better than every baseline in Table 1 on both datasets.

Fine-tuning increases Hit@1 by 0.8 points and Hit@10 by 4.0 points on NPLIB1. On MassSpecGym, it increases Hit@1 by 2.4 points and Hit@10 by 5.3 points. The main reason is that the fine-tuned policy reaches more targets. We will add the new MassSpecGym result to Table 5.

Q3. The target among the starting candidates

We ran the test searches again with the models and the settings of the paper. We changed only one thing. The target is no longer removed from the five PubChem starts. The table below shows the results in percent.

SettingNPLIB1 Hit@1NPLIB1 Hit@10NPLIB1 reachMassSpecGym Hit@1MassSpecGym Hit@10MassSpecGym reach
Target excluded (paper)29.6450.8157.7830.2241.3346.88
Target allowed34.3761.3973.2335.0452.7366.52

As you expected, GEMS performs best when the target can be among the starts.

We also looked at whether the search makes unnecessary edits. The search never removes a starting candidate. Therefore, a correct start always stays in the final candidate pool.

We then looked at the queries where the top-ranked start is the target. In 79% of these queries on NPLIB1 and in 75% on MassSpecGym, the target stays at rank 1 after the search. In most of the other queries, the search creates a structure that is not in PubChem, and this structure fits the fingerprint predicted by MIST better than the true structure does. So this effect comes from the final ranking with imperfect fingerprint predictions, which we discuss in Section 5.6. It does not come from the navigation policy. The consensus term is also not the reason. When we rank only by the fingerprint score, the target loses rank 1 in even more queries.

Our main experiments use the harder setting, where the target is not in the database and every correct answer needs at least one edit. We will add this table to the appendix.

Minor points

Thank you for these suggestions. We will define the clip function where it first appears in Equation 2, as clip(x, a, b) = min(max(x, a), b). We will also use your sentence for the retrieval step. The new text will be "We rank same-formula PubChem molecules by s(m; q), ignoring stereochemistry, and select up to K = 5 starting structures." We will also look for similar sentences in the rest of the paper and make them simpler.

Thank you, all our problems have been addressed. We changed our score to 4 (clear accept).

Reviewer_USxpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GEMS, a framework for identifying molecular structures from mass spectra and known molecular formulas by editing retrieved or generated candidate molecules. A learned policy uses predicted fingerprints and their local correspondence with the current molecule to score edits that preserve the molecular formula and valence. The policy is trained on shortest edit paths and applied in a multi-start beam search, followed by fingerprint-based and consensus ranking. GEMS achieves state-of-the-art Hit@1 and Hit@10 results on NPLIB1 and MassSpecGym while being substantially faster than recent de novo refinement baselines.

Critical strengths
  • The experiments are comprehensive, and GEMS achieves strong results on both NPLIB1 and MassSpecGym.
  • The motivation is clear. When a retrieved or generated candidate is close to the correct structure but still contains some errors, refining that candidate is more efficient than starting over from scratch.
Critical weaknesses
  • The pretraining setup excludes exact evaluation structures, but the paper does not report how similar the remaining training molecules are to the test targets. The near-analog exclusion only changes the starting candidates at inference time, while the pretrained policy remains unchanged. Therefore, it is still unclear how well GEMS generalizes to targets without close analogs in the training data.
  • The final candidate pool excludes valid proposals that were pruned from the beam, even though these proposals are still marked as visited. This means that a discarded structure cannot be reconsidered if it is reached again through a different path. The paper does not report how often this causes the correct target to be lost.
Questions for authors
  1. Could you quantify the reachability limitations of the 2-switch edit set?
  2. Is it possible to incorporate the observed spectrum or fragment-level peak evidence directly into edit selection, rather than relying only on the predicted fingerprint for spectral guidance?
  3. Could you separately ablate the local fingerprint-structure correspondence and the supervision over all optimal next actions?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer USxp, part 1 of 2

Thank you for your review. We ran new experiments for W2, Q1 and Q3. For W1, we counted the structures in the training data. Our answers are below.

W1. The similarity between the training molecules and the test targets

We have three results for this point.

First, GEMS does not need PubChem pretraining to be better than the baselines. Table 5 of the paper includes policies that are trained from scratch. These policies see only the training targets of each benchmark and the PubChem isomers that are one to three edits away from these targets. The table below shows the number of distinct structures in this training data and the test results.

Policy trained from scratchDistinct training structuresHit@1 (%)Hit@10 (%)
NPLIB1216,22128.0247.45
MassSpecGym845,53829.1439.46

We checked that none of these training structures is a validation structure or a test structure. Both policies are better than the strongest baseline in Table 1. On NPLIB1, the strongest baseline is FRIGID with 25.03% Hit@1 and 33.37% Hit@10. On MassSpecGym, the strongest baseline is our reproduction of MS-GPT with 23.23% and 27.70%. PubChem pretraining adds only 1.6 Hit@1 points on NPLIB1 and 1.1 points on MassSpecGym.

Second, the MassSpecGym split separates the molecules by structure. The split was created by clustering the molecules with the MCES distance. As a result, every test molecule has an MCES distance of at least 10 to every training molecule (Bushuiev et al., 2024). The MassSpecGym policy that is trained from scratch uses only these training molecules as targets. Its other training structures are PubChem isomers that are at most three edits away from one of these training molecules. Therefore, its training data stays close to the training molecules of the benchmark. This policy still reaches 29.14% Hit@1.

Third, this training data is much more restricted than the training data of the baselines. MS-GPT is pretrained on 100 million molecules, and FRIGID-base is pretrained on up to 230 million molecules. Like GEMS, they exclude only the exact test structures. Their training sets are between about 120 and 1,060 times larger than the training data of the GEMS policies that are trained from scratch. Therefore, we do not think that similar molecules in the training data explain the advantage of GEMS.

We also tested GEMS when close structures are removed from the starting candidates. In a new experiment for Reviewer RZko, we remove every starting candidate that has an MCES distance below 4 to the target. GEMS then reaches 26.53% Hit@1 on NPLIB1 and 24.06% on MassSpecGym. Both values are above every baseline in Table 1.

In the revision, we will state in Section 6 that the pretraining data excludes the exact validation and test structures, and that it might still contain similar structures.

W2. Targets that are lost because of beam pruning

We ran the searches of the paper again and kept every proposal. We used the same inputs, the same code and the same settings. On NPLIB1, the new run gives exactly the same candidate pool as the paper.

We counted the queries where the target was proposed during the search but was pruned from every beam. This happened in 43 of the 803 NPLIB1 queries (5.4%) and in 330 of the 17,556 MassSpecGym queries (1.9%).

Each starting candidate has its own visited set. Therefore, a target that is pruned in the search from one start can still be kept in the search from another start. This happened for 91 of the 464 NPLIB1 targets that are in the final pool.

We also tested a version of GEMS that ranks every proposal and not only the states that stay in the beam. This version has almost no extra cost, because the proposals are already constructed and their fingerprints are already computed. The table below shows the results in percent.

Candidates that are rankedNPLIB1 Hit@1NPLIB1 Hit@10NPLIB1 reachMassSpecGym Hit@1MassSpecGym Hit@10MassSpecGym reach
Beam states only (paper)29.6450.8157.7830.2241.3346.88
Every proposal30.1452.4363.1430.3141.5948.76

All metrics improve a little. Most of the recovered targets have a low position in the ranking. Their median rank is 37 on NPLIB1 and 39 on MassSpecGym. This shows that the final ranking limits the accuracy more than the pruning does. This agrees with Section 5.6. We will add these numbers and this version of GEMS to the appendix.

AuthorsOpenReview ↗

Reply to Reviewer USxp, part 2 of 2

Q1. The reachability limits of the 2-switch edits

There are two cases where a start cannot reach a target.

The first case comes from the valence of each atom, which is its number of bonds. Our edits move bonds, but they never change the valence of an atom (Proposition 1 in Appendix A.1). Here, a double bond counts as two bonds, and bonds to hydrogen atoms also count. In graph terms, the valence is the degree of the node. For most elements, this is not a restriction, because an atom of that element has the same number of bonds in every molecule in our search. For example, a carbon atom always has four bonds, a nitrogen atom has three bonds, and an oxygen atom has two bonds. Sulfur and phosphorus are different. A sulfur atom can have two, four or six bonds, and a phosphorus atom can have three or five bonds. If the sulfur atom of a start has six bonds, it still has six bonds after every edit. Therefore, this start cannot reach a target where the sulfur atom has two bonds.

The second case comes from charges. Our search uses only molecules where every atom is neutral. Some molecules can be written only with charged atoms. Examples are molecules with a nitro group or an azide group. The search cannot produce these molecules, so no start can reach such a target.

We then checked every pair of a start and a target in the main experiments. On NPLIB1, 3.2% of the pairs fall into one of these two cases, so the start cannot reach the target. On MassSpecGym, this is the case for 6.6% of the pairs. All other pairs can be connected. These are 3,763 pairs on NPLIB1 and 79,015 pairs on MassSpecGym. For each of them, a solver constructed an edit path, and we verified the path by applying its edits one by one. No pair remains undecided.

For 95.4% of the NPLIB1 queries and 95.2% of the MassSpecGym queries, at least one of the five starts can reach the target. This is another reason why GEMS starts the search from several candidates. A single start can fall into one of the two cases above, but it might be enough that one of the starts can reach the target. The queries without such a start count as failures in all of our results.

These results show that reachability is not the main limit of a search with three edit rounds. The main limit is the distance between the start and the target. With exact fingerprints, 100 random starts recover 89.91% of the NPLIB1 targets within ten edit rounds (Table 7). We will add this analysis to the appendix.

Q2. Using the spectrum or the peaks directly for edit selection

Yes, this is possible. At the moment, the information from the peaks enters GEMS through MIST, which reads the spectrum and the formula annotations of the peaks. We give only the predicted fingerprint to the policy for two reasons.

The first reason is the training data. With fingerprint guidance, the policy can be trained on structures alone. We can compute the shortest edit paths and the exact fingerprints for any pair of isomers. This makes the pretraining on PubChem possible. If the policy also needed the peaks as input, we could train it only on molecules that have measured spectra.

The second reason is speed. The policy scores all valid edits of a molecule in one forward pass. Methods that use predicted spectra to guide the refinement must predict a spectrum for every candidate.

Spectra already help GEMS in the ranking step. When we rerank the top ten candidates of GEMS with spectra that are predicted by ICEBERG, Hit@1 increases by 2.24 points on NPLIB1 and by 1.38 points on MassSpecGym (Appendix J.3).

Using the peaks directly for edit selection is a good direction for future work. For example, the peaks could be added as extra input tokens and connected to the substructures that can explain them. We will add this point to the discussion section.

Q3. Separate ablations of the correspondence and of the supervision

We ran both ablations on NPLIB1 with three random seeds for each model. This is a short summary. The full answer, with the setup and the tables, is in our reply to Reviewer WAfP.

The correspondence helps. In the ablation, the policy still receives all fingerprint tokens, but they are not connected to atoms and bonds. Hit@1 then falls from 27.48% to 26.28%, and Hit@10 falls from 47.57% to 45.29%. The 95% confidence intervals of both differences exclude zero. We see the same result at every search width from 1 to 16.

Training with one optimal action gives results that are similar to training with all optimal actions. Hit@1 is 28.06% when the selected edit is fixed, 27.94% when a new edit is selected at every visit, and 27.48% with all optimal actions. The confidence intervals of these differences include zero. This result shows that the important part is the shortest-path supervision, whose labels are computed from molecular structures only. We will update the text of the paper to say this, and we will add these results to the appendix.

Reviewer_WAfPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GEMS addresses molecular structure elucidation from MS/MS spectra and a known molecular formula through learned search over isomer graphs. It uses formula- and valence-preserving bond edits, a policy that connects predicted fingerprint features to candidate atoms and bonds, and shortest-path supervision derived from molecular structures. Multiple searches are combined through fingerprint scoring and consensus ranking. Experiments on NPLIB1 and MassSpecGym show improved identification accuracy, efficient search, and successful refinement of generated candidates. Oracle experiments separately examine fingerprint errors in navigation and ranking.

Critical strengths
  1. GEMS improves identification on both benchmarks with the target excluded from retrieval. It also outperforms FRIGID’s refinement when starting from the same initial generated pool, supporting its usefulness beyond PubChem initialization.

  2. The enumeration baseline, exclusion of one-edit neighbors, and separate oracle interventions for navigation and ranking provide useful evidence about what drives performance. The efficiency gains over enumeration also demonstrate the practical value of learned proposals.

Critical weaknesses
  1. Tables 5–6 establish the effectiveness of the complete system but do not compare against an otherwise matched policy without explicit fingerprint-to-atom/bond correspondence. Likewise, the REINFORCE comparison does not establish the benefit of supervising all optimal actions over choosing one. These omissions leave the contributions of two central design choices unclear.

  2. The runtime comparison uses different timing boundaries. FRIGID includes per-round evaluation, while GEMS excludes post-hoc evaluation, limiting interpretation of the headline speedup.

Questions for authors
  1. Under matched training and search settings, how does performance change when the fingerprint correspondence relations are removed, and separately when supervision uses only one optimal action?

  2. What is FRIGID’s runtime after excluding ground-truth evaluation while retaining the generation, spectral prediction, and ranking required for inference?

1 comment

Thank you for your review. We ran the two ablations that you asked for. We also measured the runtime of FRIGID again.

W1 and Q1. The fingerprint correspondence and the supervision with one optimal action

We trained every model with the from-scratch NPLIB1 recipe of the paper. All models use the same data, the same 34,476 updates and the same learning rate schedule. We trained three random seeds for each model. For a given seed, all models start from the same initial weights. We evaluate every model after the last update, so all models are compared at the same point of training. The search settings are the same as in the paper. We use K=5, P=B=16, D=3 and λ=16 on all 803 test queries.

We compare GEMS with three other models.

  1. The model without correspondence does not know which atoms and bonds generate each fingerprint bit. To remove this information, we fix the attention biases of Section 4.3 at zero. This model still receives all fingerprint tokens.
  2. In the two models with one optimal action, each training state has only one optimal edit as its label. GEMS instead maximizes the total probability of all optimal edits (Equation 4). In the first of these two models, the selected edit stays the same for all epochs. In the second model, we select a new optimal edit every time the state is visited.

The table below shows the test results in percent. Each value is the mean and the standard deviation over the three seeds. The last two columns show the difference to GEMS, with a paired bootstrap 95% confidence interval.

ModelHit@1Hit@10Difference in Hit@1Difference in Hit@10
GEMS27.48 ± 0.8347.57 ± 0.45
No correspondence26.28 ± 0.5445.29 ± 0.64-1.20 [-2.36, -0.08]-2.28 [-3.58, -1.02]
One optimal action, fixed28.06 ± 0.3847.82 ± 0.57+0.58 [-0.37, +1.54]+0.25 [-0.80, +1.32]
One optimal action, redrawn27.94 ± 0.3147.53 ± 0.76+0.46 [-0.38, +1.30]-0.04 [-1.03, +0.95]

The 28.02% in Table 5 comes from one seed at a checkpoint that was selected on the validation set. For this reason, it is a little different from the GEMS value in this table.

The correspondence helps. When we remove it, the accuracy becomes lower at every search width that we tested. The next table shows Hit@1 in percent for search widths from 1 to 16.

Search width P=B124816
GEMS19.1823.0425.5726.6927.48
No correspondence17.1421.3423.5424.9926.28

Without the correspondence, Hit@1 is between 1.2 and 2.0 points lower, and Hit@10 is between 2.3 and 2.9 points lower. All of these confidence intervals exclude zero. Both models receive the same fingerprint tokens. Therefore, the improvement comes from the connection between the fingerprint bits and the atoms and bonds.

For the supervision, training with one optimal action gives results that are similar to training with all optimal actions. The differences are small, and all four confidence intervals include zero. This result shows that the important part is the shortest-path supervision. Its labels are computed from molecular structures only, and it is better than REINFORCE in a matched comparison (28.02% against 24.78% Hit@1 in Table 5). We will update the text of contribution 3 and of Section 4.4 to say this. We will also add both tables to the appendix.

W2 and Q2. The runtime of FRIGID without ground-truth evaluation

We first measured how much of the FRIGID runtime in the paper was spent on ground-truth evaluation. It was 164 seconds out of 94,408 seconds, which is 0.17% of the total. Without the evaluation, the FRIGID runtime is 94,244 seconds.

We also ran FRIGID again to collect more accurate timing information. In the new run, the evaluation runs in a separate process, and we measure the time of every phase.

The table below shows the runtimes on all 803 NPLIB1 test spectra. The last column is the FRIGID time divided by the GEMS time.

MeasurementFRIGID (seconds)GEMS (seconds)Ratio
Run in the paper, as reported94,408200472
Run in the paper, without evaluation94,244200471
New run without evaluation, from process start to process end92,657200463
New run, processing time only88,945165.6537

The last row counts only the processing time for both methods. It includes generation, spectrum prediction and ranking. It does not include the start of the process, the loading of the models and the data, or the writing of the output.

We will add this table to Appendix K.

Comments on the paper

1 comment
Area_Chair_Ui7AOpenReview ↗

Author-reviewer discussion reminder

Dear Reviewers,

Please spend some time reading the rebuttal and indicate whether the authors have clarified your concerns. Please do so before Nov 18 AoE to give the authors the chance to further clarify.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.