AMMAS: Multimodal Modeling for Atomistic Science with 3D Structures and Language

In atomistic science, the spatial arrangement of atoms determines the properties and interactions of microscopic 3D structures. Natural language provides a complementary interface for expressing structural concepts, questions, and desired geometric changes. Coupling these modalities would enable geometry-grounded 3D structure understanding and language-guided 3D structure generation. However, a modality gap remains: language modeling operates over discrete token sequences, whereas geometric modeling operates over continuous atomic coordinates. To bridge this gap, we propose A Multimodal Model for Atomistic Science (AMMAS), which supports autoregressive generation of both discrete text and continuous 3D structures. AMMAS interleaves stacks of Shared Generalist Transformer (SGT) layers with Modality Expert Transformer (MET) layers. SGT layers jointly update the mixed sequence of atomistic and textual representations, enabling cross-modal interaction. MET layers then use modality-specific positional encoding and attention patterns to update the text-contextualized atomistic representations of each 3D structure. The updated atomistic representations feed into subsequent SGT layers, establishing a multimodal latent co-evolution flow across depth. We instantiate AMMAS for small molecules and introduce three benchmarks to evaluate geometry-grounded understanding and language-guided generation: conformational question answering, stereochemical comparison, and text-guided conformation editing. Across comparisons with eight learned baselines, AMMAS achieves the best overall performance on all benchmarks and ranks first in 14 of 15 subcategories. Against the strongest baselines, it improves the two understanding accuracies by 8.95 and 10.46 percentage points, and editing performance by 24.96 points in V-Success@3 Å and a relative 75.4% in V-NETP.