TAG-XMM: Multimodal and Multi-Model Joint Multispectral Representations for Molecular Structure Elucidation

Molecular structure elucidation reconciles complementary spectroscopic measurements that each inform on part of a chemical structure: ${}^1$H and ${}^{13}$C NMR on proton/carbon environments, HSQC on which proton sits on which carbon, IR on functional groups and MS/MS on fragments. We introduce Token-Aggregated Generation Across Models and Modalities (TAG-XMM), a formula-conditioned framework that jointly encodes these five analytical spectra into a chemically supervised token bank, which a resampler condenses into memory for an autoregressive (AR) decoder whose beam-search candidates form a pool of candidate structures. To increase robustness to cases where the correct structure is absent from the beam, we introduce the Graph-SSI (Graph Spectrum-Steered Inpainting) mechanism. In Graph-SSI, a learned gate estimates the probability that the true structure is missing from the pool. Small regions where the candidates disagree are located, and their atom states and bond orders are regenerated by bond swaps and a discrete graph diffusion model conditioned on the encoded spectra. Finally, a verifier adds to the pool the few graphs that best fit the observed spectra. To select the answer, an inverse solver ranks the resulting augmented pool by the decoder's score for each candidate, as well as by comparing fit of forward-predicted spectra for each candidate to the initial spectra. Our results show that on a large simulated corpus, the decoder reaches 77.88% Exact@1 on 39,224 test molecules. On all 39,224 test molecules, all components provide additive performance benefits and the final pipeline reaches 90.03% Exact@1 accuracy and Graph-SSI corrections push performance to 91.19%, ranking first 519/2,406 structures the beam never proposed; on each baseline's own spectra, TAG-XMM still leads (86.26% vs 80.02% for NMRViT and, without the formula as in NMIRacle, 72.07% vs 55.63%). Code is provided at https://anonymous.4open.science/r/TAG-XMM-6966