acceptodds
Under review as a conference paper at ICLR 2027

TAG-XMM: Multimodal and Multi-Model Joint Multispectral Representations for Molecular Structure Elucidation

Abstract

Molecular structure elucidation reconciles complementary spectroscopic measurements that each inform on part of a chemical structure: H and C NMR on proton/carbon environments, HSQC on which proton sits on which carbon, IR on functional groups and MS/MS on fragments. We introduce Token-Aggregated Generation Across Models and Modalities (TAG-XMM), a formula-conditioned framework that jointly encodes these five analytical spectra into a chemically supervised token bank, which a resampler condenses into memory for an autoregressive (AR) decoder whose beam-search candidates form a pool of candidate structures. To increase robustness to cases where the correct structure is absent from the beam, we introduce the Graph-SSI (Graph Spectrum-Steered Inpainting) mechanism. In Graph-SSI, a learned gate estimates the probability that the true structure is missing from the pool. Small regions where the candidates disagree are located, and their atom states and bond orders are regenerated by bond swaps and a discrete graph diffusion model conditioned on the encoded spectra. Finally, a verifier adds to the pool the few graphs that best fit the observed spectra. To select the answer, an inverse solver ranks the resulting augmented pool by the decoder's score for each candidate, as well as by comparing fit of forward-predicted spectra for each candidate to the initial spectra. Our results show that on a large simulated corpus, the decoder reaches 77.88% Exact@1 on 39,224 test molecules. On all 39,224 test molecules, all components provide additive performance benefits and the final pipeline reaches 90.03% Exact@1 accuracy and Graph-SSI corrections push performance to 91.19%, ranking first 519/2,406 structures the beam never proposed; on each baseline's own spectra, TAG-XMM still leads (86.26% vs 80.02% for NMRViT and, without the formula as in NMIRacle, 72.07% vs 55.63%). Code is provided at https://anonymous.4open.science/r/TAG-XMM-6966

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
de novo molecular generation, representation learning, spectroscopy, multimodality, chemistry, autoregressive decoding, discrete graph diffusion, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1RC9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TAG-XMM is a formula-conditioned molecular elucidation system combining ¹H NMR, ¹³C NMR, HSQC, IR, and MS/MS to recover exact molecular connectivity without stereochemistry. A chemically supervised multimodal encoder produces a token bank, which a resampler uses to condition an autoregressive SMILES decoder and generate a candidate pool. Graph-SSI uses confidence and spectral-fit features to decide whether to repair a pool, selects local regions using bond disagreement among candidates, and proposes new graphs through bond swaps and spectrum-conditioned discrete graph diffusion.

Critical strengths
  1. The paper separates missing-candidate failures from ranking failures and provides a concrete spectrum-conditioned repair mechanism for the former. When the true structure is absent from the beam, reranking that pool cannot recover it.
  2. Detailed diagnostics and controls reveal where the system improves and fails.
Critical weaknesses
  1. Some mass differences can be calculated from observations and a formula, but compositions assigned using the true structure cannot automatically be treated as available from experimental spectra alone. The paper lacks retraining results without these fields and does not establish that equivalent features can be obtained without accessing the unknown structure.
  2. Important design claims remain confounded, and the necessity of the complex system’s individual mechanisms is unclear. The 97.85% versus 73.41% joint-encoding/late-fusion comparison is a central argument, but Appendix C explicitly states that the encoder was trained only on all five modalities, while late fusion combines its out-of-distribution single-modality outputs. This mixes fusion design with training-input distribution and cannot establish the same advantage under fair training.
  3. Table 11’s 90.04% variant without two in-domain energies further shows that gains along one sequential build-up do not establish each scoring module’s necessity. Fairly trained fusion baselines and analyses separating newly recovered structures, pool-standardization effects, and additional search computation would clarify the methodological advance relative to the system’s resource investment.
Questions for authors
  1. For each MS/MS field and related summary descriptor in Table 4, which are determined solely by observed spectra and the permitted molecular formula, and which read true-structure simulator annotations? After removing the latter and retraining the affected components, or predicting them with an independent spectral annotation model, what are the stage-wise full-system Exact@1 results?
  2. Beyond the disclosed shift-head overlap, what are the test-molecule overlaps with the encoder, MolForge decoder pretraining, diffusion model, and forward predictors? What are the final Graph-SSI results and gained/lost counts on a rigorously defined non-overlapping subset?
  3. Which Test A metrics had been inspected before adopting the isotonic rule? Is there a new frozen evaluation unused in any design decision that confirms the final rule’s gain over the inverse solver?
  4. How much of the joint-encoding advantage remains when late fusion uses adequately trained single-modality encoders, or when both approaches use the same modality-subset training?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1RC9

We thank Reviewer 1RC9 for the detailed review. References below use the revised manuscript, with changes marked in blue.

W1/Q1: MS/MS provenance and stage-wise retraining. Table 4 now distinguishes observed quantities (m/zm/z, intensities and their summary), the formula-dependent precursor fraction, and annotation-derived neutral loss, parity, fragment/loss compositions and 11 descriptors. The other 407 descriptors use no MS/MS. We retrained the encoder and decoder with annotation-derived fields retained, removed (including their fill value and all 11 descriptors) or recomputed from masses and formula alone (App. G, Table 14).

MS/MS fieldsBest stringMolecule marginalSolver, allSolver, frozenTAG-XMM, frozenGains/losses
Retained78.81%82.17%90.30%87.34%88.10%80/30
Removed78.33%81.84%89.59%86.38%87.07%73/27
Recomputed79.08%82.62%90.04%87.57%88.58%95/28

The first three accuracy columns use all 39,224 molecules; the remaining columns use the frozen 6,653, with gains/losses relative to each variant's solver. Data, schedule and seed are matched; the gate, verifier, isotonic map and solver weights remain fixed. Recalculation assigns the nearest ion subformula within max⁡(10 ppm,2 mDa)\max(10\,\mathrm{ppm}, 2\,\mathrm{mDa}), agreeing with the simulator on 629,552/629,561 jointly assigned peaks across 12,720 simulated molecules. Removal reduces solver accuracy (p=6.7×10−7p=6.7\times10^{-7}); recalculation recovers most of the difference (p=0.074p=0.074 versus retained fields).

W2/Q4: fusion under matched training. Figure 3 replaces the confounded comparison with independently trained single-spectrum encoders and joint encoders using the same 715,840 records, seed and schedule. All-five joint encoding reaches 97.66% versus 93.99% for late fusion: +3.66 points [2.96, 4.35]. Spectrum-subset training gives 97.39%, or +3.39 [2.68, 4.08]. Its average advantages with three and four spectra are +1.64 [1.20, 2.12] and +2.80 [2.27, 3.37] points; late fusion leads with one or two (App. C, Table 8a).

W3: new candidates, reranking and module necessity. Graph-SSI's net gain of 456 consists of 519 out-of-beam recoveries and 63 net in-beam losses. Added graphs displace 126 correct answers; reranking among original candidates gains 135 and loses 72 (p=1.4×10−5p=1.4\times10^{-5}; App. F).

Each post hoc solver ablation was refitted on validation (App. E, Table 11a):

Removed view(s)Exact@1 change, pointsExact McNemar pp
Decoder marginal−1.486.5×10−706.5\times10^{-70}
IR−0.425.2×10−105.2\times10^{-10}
DP4-style 13^{13}C likelihood−0.288.0×10−78.0\times10^{-7}
HSQC register−0.192.9×10−32.9\times10^{-3}
MS/MS−0.121.9×10−31.9\times10^{-3}
External 1^{1}H, 13^{13}C, HSQC and both in-domain energies together−0.321.0×10−71.0\times10^{-7}

The five views grouped in the last row each change accuracy by at most 0.09 points when removed individually, supporting partial redundancy. Wider search is also effective: width 320 plus the solver recovers 940 originally out-of-beam structures, while missing 139 of Graph-SSI's 519 recoveries (Table 10a). Graph-SSI takes 4.68 seconds per pool sent on one H100, for 34.9% of pools.

Q2: component overlaps. Table 2a reports constitution overlaps with encoder/decoder training (1,340 records), IR fine-tuning (418), shift heads (2,076), diffusion (0), published HSQC lists (262), and the list distributed with MolForge (3,643). Excluding constitution or first-InChIKey-block matches with the training data of the four models we trained leaves 35,983 molecules: TAG-XMM reaches 90.63%, with 645 gains and 194 losses over the inverse solver (p=2.8×10−57p=2.8\times10^{-57}). Further excluding the MolForge-associated list leaves 32,804 molecules and 595 gains/172 losses (p=2.4×10−55p=2.4\times10^{-55}). That list is not a verified pretraining manifest.

Q3: inspected results and frozen confirmation. We had inspected all Test A results, including the isotonic sensitivity analysis, controls, secondary analyses and test-label diagnostics, before adopting isotonic scoring. Both that choice and the move to the inverse solver were test-informed; the map itself was fitted on validation. Test A's primary comparison, using the offset rule and six-view solver, was null: 510 gained, 491 lost on 32,571 molecules (p=0.57p=0.57). We therefore scored the 6,653 test molecules Test A did not cover, after every setting was fixed, once with the final system: Graph-SSI gains 134 and loses 53 over the inverse solver (88.83% vs 87.61%; p=2.8×10−9p=2.8\times10^{-9}). Within the strict subset and outside the first 2,048 test molecules, on which the MS/MS minimum-peak setting was chosen, it gains 118 and loses 49 (p=9.5×10−8p=9.5\times10^{-8}), and the gain recurs in all three retrains above (80/30, 73/27, 95/28).

Thank you for the revision. I maintain my score, because the authors disclose that all Test A results were inspected before adopting isotonic scoring and the inverse solver, and their pre-specified primary comparison was null, so the 91.19% headline is not an estimate of expected performance on unseen data, and no end-to-end evaluation on measured spectra exists.

Reviewer_ED67OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAG-XMM reconstructs molecular connectivity from five spectroscopic modalities and a molecular formula. It combines a joint spectral encoder, an autoregressive SMILES decoder, a local graph-repair module (Graph-SSI), and spectrum-based candidate reranking.

On a simulated benchmark, the complete pipeline improves Exact@1 from 77.88% to 91.19%. While the system is comprehensive, concerns about privileged input features, test independence, baseline fairness, and practical generalization limit the strength of its conclusions.

Critical strengths
  1. The framework clearly distinguishes candidate-generation failures from ranking failures and introduces dedicated components to address each.
  2. The encoder incorporates chemically motivated supervision and explicit correspondence between HSQC and one-dimensional NMR signals.
  3. The paper provides detailed error analyses, paired statistical comparisons, and controls using mismatched spectra to investigate the sources of improvement.
Critical weaknesses
  1. Privileged input information. Table 4 states that several MS/MS features derive from simulator annotations of the true structure. Without retraining after removing these features, the reported accuracy does not establish performance using independently observable spectral inputs.
  2. Insufficient test independence. The training data of the in-domain shift predictors overlap with 2,082 test records, while the overlap-free analysis covers only the reranker.
  3. Unfair fusion and baseline comparisons. The late-fusion baseline uses single-modality inputs with an encoder trained exclusively on complete multimodal inputs, placing it out of distribution. External comparisons also use unequal training-data sizes, preventing a clean attribution of the gains to the proposed method.
  4. Limited evidence of practical generalization. All evaluations use simulated spectra, and the deployed model assumes complete modalities and an accurate molecular formula. Performance under experimental spectra with noise and uncertainty remains unestablished.
  5. Unclear necessity and computational value of the complex pipeline. Graph-SSI adds 1.16 percentage points but introduces multiple generation and verification stages without runtime or compute-matched comparisons.
Questions for authors
  1. How does the complete pipeline perform when all MS/MS features derived from true-structure annotations are removed during both training and evaluation?
  2. Can the authors evaluate the frozen, complete system on a fresh test set with no overlap with any component’s training data?
  3. Does joint encoding still outperform late fusion when the single-modality encoders are properly trained and training-data sizes are matched?
  4. How robust is the system to experimental spectra?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ED67

We thank Reviewer ED67 for the detailed review.

The revision adds the requested controls, marked in blue. References below use the revised manuscript, with changes marked in blue.

W1/Q1: annotation-derived MS/MS inputs. Table 4 identifies neutral loss, parity, fragment/loss compositions and 11 summary descriptors as annotation-derived. We retrained the encoder and decoder with these fields retained, removed during both training and evaluation, or recomputed solely from peak masses and the formula (App. G, Table 14):

MS/MS fieldsSolver, all 39,224Solver, frozen 6,653TAG-XMM, frozen 6,653Graph-SSI gains/losses
Retained90.30%87.34%88.10%80/30
Removed89.59%86.38%87.07%73/27
Recomputed90.04%87.57%88.58%95/28

Gains/losses compare each variant with its own solver on the frozen cohort. The retrainings use the same data, schedule and seed; downstream scoring components remain fixed. Removing the fields reduces solver accuracy (p=6.710-7); mass-based assignment recovers most of the difference (p=0.074 versus retained fields). It uses the nearest ion subformula within (10ppm,2mDa), matching the simulator on 629,552/629,561 jointly assigned peaks across 12,720 molecules. Graph-SSI improves all three variants. Validation on measured MS/MS remains open.

W2/Q2: test independence. Appendix A now audits overlaps by component and reports complete-system results. The strict subset excludes matches by constitution or first InChIKey block with encoder/decoder training, IR fine-tuning, shift-head training and diffusion training. The frozen cohort contains previously unpooled test records evaluated once after all settings were fixed.

CohortMoleculesGraph-SSI gains/lossesExact McNemar p
Strict subset35,983645/1942.810-57
Frozen cohort6,653134/532.810-9
Frozen ∩ strict, excluding the first 2,048 test records5,538118/499.510-8
Scaffold absent from training6,517217/487.710-27

TAG-XMM reaches 90.63% on the strict subset; the scaffold-novel gain is 2.59 points (Fig. 7a). These are analyses of the existing test split. A fresh test set independent of every component, including external pretraining, remains unavailable. The isotonic rule was selected after inspecting the earlier test run, and frozen-cohort decoder accuracy had already been measured. Decoding also followed one optimizer step on test SMILES: omitting it changes correct solver predictions from 35,314 to 35,315; Graph-SSI has not been rerun for that control. The first 2,048 records were used to choose the MS/MS minimum-peak threshold, hence their exclusion above.

W3/Q3: fair fusion and training size. Figure 3 now compares encoders trained on the same 715,840 records, seed and schedule. All-five joint encoding reaches 97.66% same-formula retrieval versus 93.99% for late fusion of independently trained single-spectrum encoders: +3.66 points [2.96, 4.35]. A joint encoder trained on spectrum subsets reaches 97.39%; its average advantages with three and four spectra are +1.64 and +2.80 points. Late fusion leads with one or two spectra (App. C, Table 8a), so the revised claim is specific to the supported settings.

External training sizes remain unmatched. On NMRViT’s inputs, our decoder scores 75.03% versus 80.02% for NMRViT, increasing to 86.26% after ranking and repair (Table 1c). This identifies where our pipeline improves, but a training-size-matched rerun remains planned.

W4/Q4: experimental spectra and incomplete inputs. We added an encoder-only evaluation on measured literature 1^1H+13^{13}C spectra: 54.03% same-formula retrieval on 4,096 molecules in 984 families, versus 24.02% chance, with no constitution or first-InChIKey-block overlap with the training split (App. B). Full-pipeline validation under experimental noise and formula uncertainty remains future work. On simulated reduced-input benchmarks, decoder retraining gives 86.26% versus NMRViT’s 80.02%, and 72.07% versus NMIRacle’s 55.63% without a formula; Graph-SSI was not run in the latter condition.

W5: Graph-SSI’s value and computational cost. Graph-SSI recovers 519 structures absent from beam 40, offset by 63 net in-beam losses, for 456 additional correct answers overall. It takes 4.68 seconds per pool sent to repair on one H100; the gate sends 34.9% of pools (App. F). At beam width 320, the inverse solver recovers 940 originally out-of-beam structures, but misses 139 of Graph-SSI’s 519 recoveries (App. D, Table 10a). Larger beams therefore recover more overall, while repair contributes distinct answers. This post hoc comparison omits the test-SMILES update for wider beams and does not match computation; it establishes complementary recovery, not computational superiority.

Reviewer_6rPCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TAG-XMM, a multimodal framework for molecular structure elucidation that integrates five types of spectroscopic data using a joint encoder, an autoregressive SMILES decoder, graph-based structure refinement, and spectral-consistency reranking. The proposed Graph-SSI module combines bond swaps and spectrum-conditioned graph diffusion to recover molecular structures missing from the initial candidate pool. Evaluated on 39,224 molecules with simulated spectra, the framework achieves 91.19% Exact@1 accuracy, demonstrating the effectiveness of combining multimodal spectral information with candidate generation and refinement for molecular structure elucidation.

Critical strengths
  1. Effective graph-based refinement of molecular structures. The proposed Graph-SSI module addresses an important limitation of candidate reranking by recovering correct molecular structures that are absent from the initial beam. It successfully recovers and ranks first 519 of the 2,406 out-of-beam molecules, demonstrating its ability to correct errors that cannot be addressed by reranking alone. The combination of rule-based bond swaps and spectrum-conditioned graph diffusion is well motivated, and the experiments demonstrate their complementary contributions to structure recovery. This provides a meaningful improvement beyond conventional candidate generation and reranking.
  2. Strong performance improvements over existing methods. TAG-XMM achieves 91.19% Exact@1 accuracy on the simulated benchmark and demonstrates improvements over NMRViT and NMIRacle on matched test subsets under their respective spectral input conditions. The component-wise analyses further show how candidate reranking and graph refinement contribute to the overall performance. These results support the effectiveness of the proposed framework.
Critical weaknesses

The multimodal encoder is trained with seven objectives incorporating different forms of chemical supervision. However, their individual contributions to representation quality and downstream molecular structure prediction are not systematically evaluated. This makes it difficult to assess how the proposed chemical supervision contributes to the final structure elucidation performance and which objectives are most important to the reported improvements.

Questions for authors
  1. MS/MS input features. Appendix B indicates that some MS/MS features, including fragment and neutral-loss compositions, are derived from simulator annotations of the true molecular structure. Could the authors clarify whether these features can be obtained independently from experimental mass spectra, and to what extent the reported performance depends on them?
  2. Choice of molecular representation for generation. What motivated the use of a SMILES-based autoregressive decoder for initial molecular generation, followed by graph-based refinement, rather than directly generating molecular graphs? Given that a beam width of 40 yields only 4.69 distinct molecular structures on average, it would be helpful to discuss the advantages and potential limitations of this design choice.
  3. Whether Gate threshold. The Whether Gate uses a threshold of 0.01, which was originally selected for an earlier replacement-based refinement strategy and retained in the final candidate-augmentation framework. Could the authors clarify the rationale for this choice? Has the sensitivity of the final accuracy and refinement performance to this threshold been evaluated?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6rPC

We thank Reviewer 6rPC for the detailed review.

References below use the revised manuscript, with changes marked in blue.

Contribution of the encoder objectives. We retrained the encoder with either within-formula alignment or HSQC–NMR correspondence removed, using the same recipe as the full seven-objective reference (App. C, Table 8b). On 4,095 test molecules in 1,223 formula families, removing within-formula alignment lowers retrieval from 97.66% to 96.90% (−0.76 points; paired 95% formula-family CI [−1.13, −0.39], p=1.210-4). Removing correspondence supervision gives 97.22% (−0.44 points [−0.78, −0.09], p=0.027) and reduces the dedicated head’s HSQC-to-1^1H/13^{13}C correspondence accuracy from 99.72%/99.71% to 42.28%/12.32% (chance: 12.56%/7.99%). These single-seed ablations support the objectives’ roles at the representation level. A complete objective-by-objective analysis remains open; the carbon-environment/join ablation and downstream structure-prediction tests are still pending.

Q1: obtaining the MS/MS features without the structure. Table 4 now identifies every field’s source. We tested a structure-independent replacement that assigns each peak the nearest ion subformula allowed by the molecular formula within max(10 ppm, 2 mDa). On 12,720 simulated molecules, it agrees with the simulator’s fragment composition on 629,552 of the 629,561 peaks both methods assign.

MS/MS annotation-derived fieldsInverse solver Exact@1p versus retained fieldsGraph-SSI gains/losses
Retained90.30%—80/30
Removed89.59%6.710-773/27
Recomputed from peak masses and formula90.04%0.07495/28

The solver is evaluated on all 39,224 molecules; Graph-SSI gains/losses compare with each variant’s own solver on the frozen 6,653-molecule cohort (App. G, Table 14). Data, schedule and seed are matched, and downstream scoring components remain fixed. The features’ contribution is largely recovered on this benchmark. Assignments can nevertheless be ambiguous: only 66.6% of peaks have a unique candidate even at exact simulated masses. Measured-MS/MS reliability remains untested.

Q2: SMILES generation followed by graph refinement. SMILES let us use a pretrained MolForge decoder and formula-constrained beam search; graphs allow direct edits to connectivity. The beam contains the truth for 93.87% of molecules. Its 40 strings represent 4.69 distinct molecules on average, but combining their probabilities improves decoder Exact@1 from 77.88% to 81.74% (Table 1a). Graph-SSI then corrects local errors: 418 of its 519 out-of-beam recoveries are one bond swap from a top-four decoder candidate.

Wider search also helps. At width 320, the inverse solver recovers 940 of the 2,406 originally out-of-beam structures, but misses 139 of Graph-SSI’s 519 recoveries (App. D, Table 10a). This supports complementary recovery; it is not a compute-matched comparison. The design’s limitations are duplicate strings consuming beam capacity and repair being restricted to local changes. A direct whole-graph generator comparison remains open.

Q3: gate threshold. The 0.01 threshold was selected on validation for the earlier replacement rule and retained for augmentation. It favors recovery: 34.9% of pools are sent to repair, including 98.0% of out-of-beam cases. Original candidates remain available, although added graphs can displace correct answers; in-beam accuracy changes from 95.92% to 95.74%.

The new post hoc sweep shows the trade-off (App. F, Table 13a). Raising the threshold from 0.01 to 0.1 reduces the fraction sent from 34.9% to 12.8%, out-of-beam recoveries from 519 to 425, and overall Exact@1 from 91.19% to 90.91% (90.03% without repair). Every tested higher threshold lowers accuracy. The deployed threshold was unchanged by this analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.