acceptodds
Under review as a conference paper at ICLR 2027

AMMAS: Multimodal Modeling for Atomistic Science with 3D Structures and Language

Abstract

In atomistic science, the spatial arrangement of atoms determines the properties and interactions of microscopic 3D structures. Natural language provides a complementary interface for expressing structural concepts, questions, and desired geometric changes. Coupling these modalities would enable geometry-grounded 3D structure understanding and language-guided 3D structure generation. However, a modality gap remains: language modeling operates over discrete token sequences, whereas geometric modeling operates over continuous atomic coordinates. To bridge this gap, we propose A Multimodal Model for Atomistic Science (AMMAS), which supports autoregressive generation of both discrete text and continuous 3D structures. AMMAS interleaves stacks of Shared Generalist Transformer (SGT) layers with Modality Expert Transformer (MET) layers. SGT layers jointly update the mixed sequence of atomistic and textual representations, enabling cross-modal interaction. MET layers then use modality-specific positional encoding and attention patterns to update the text-contextualized atomistic representations of each 3D structure. The updated atomistic representations feed into subsequent SGT layers, establishing a multimodal latent co-evolution flow across depth. We instantiate AMMAS for small molecules and introduce three benchmarks to evaluate geometry-grounded understanding and language-guided generation: conformational question answering, stereochemical comparison, and text-guided conformation editing. Across comparisons with eight learned baselines, AMMAS achieves the best overall performance on all benchmarks and ranks first in 14 of 15 subcategories. Against the strongest baselines, it improves the two understanding accuracies by 8.95 and 10.46 percentage points, and editing performance by 24.96 points in V-Success@3 Å and a relative 75.4% in V-NETP.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, multimodal understanding and generation, 3D molecular modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_iXtuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AMMAS, a multimodal framework for integrating 3D atomic structures and natural language. It interleaves two types of Transformer layers: Shared Generalist Transformers for cross-modal interaction and Modality Expert Transformers for geometry-specific modeling. This design enables geometry-grounded understanding and language-guided generation of continuous 3D structures. Experiments on three newly introduced small-molecule benchmarks demonstrate its effectiveness over existing baselines.

Critical strengths
  1. The paper presents a clear motivation and a well-structured framework. The overall methodology is clearly explained and effectively illustrated, making the proposed approach easy to follow.
  2. The paper provides comprehensive experimental evaluations across three benchmarks, with comparisons against various representative baselines. The ablation studies further support the effectiveness of the proposed architectural design.
Critical weaknesses
  1. I think the current ablations mainly focus on architectural placement, while more in-depth analyses of the co-evolution mechanism and the contribution of continued pretraining (CPT) are still missing. This is particularly important for the generation task, where AMMAS and the Qwen3 baseline differ in several aspects, making it difficult to isolate the contribution of the proposed method.
  2. Although the authors introduce three new benchmarks, they mainly focus on a relatively narrow set of capabilities, such as geometric classification and constrained conformation editing. I am not sure how well the proposed framework generalizes to broader atomistic science applications. It would also be helpful to evaluate or align the model with more widely adopted benchmarks to better demonstrate its general capabilities.
  3. While the paper reports substantial improvements in conformation editing, the advantage over BioMatrix under the stricter [email protected] A metric is only around 1 percentage points. I think this deserves further analysis, as the improvement appears much less significant than the gains highlighted elsewhere in the paper. It would be helpful to clarify where the reported performance gains mainly come from.
  4. Given that several benchmark labels are derived from explicit geometric rules, I think a non-neural geometry-based baseline would be highly informative. Without such a comparison, it is difficult to assess whether the proposed model demonstrates distinctive multimodal reasoning capabilities or mainly learns to approximate deterministic geometric computations that conventional tools can perform directly.
Questions for authors
  1. Clarify the specific contribution of the proposed co-evolution mechanism beyond the architectural placement of SGT and MET layers, and explain whether the current ablations sufficiently isolate its effectiveness.
  2. Discuss the contribution of continued pretraining (CPT) to the overall performance, particularly for the generation task, where AMMAS and the Qwen3 baseline differ in several architectural and training components.
  3. Clarify the generalizability of the proposed framework beyond the three self-constructed benchmarks, and discuss whether evaluation on existing widely adopted benchmarks is feasible.
  4. Explain why the improvement over BioMatrix under the stricter [email protected] A metric is relatively small, despite the substantial gains reported in other editing metrics.
  5. Discuss whether conventional geometry-based methods could achieve competitive performance on tasks whose labels are derived from explicit geometric rules, and clarify what additional capabilities AMMAS provides beyond such methods.
Reviewer_35a4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AMMAS, a model for 3D molecular structures and language that autoregressively generates both text and continuous atomic coordinates. Shared Generalist Transformer layers, initialized from Qwen3-4B-Base, process the mixed text–atom sequence. They are interleaved with Modality Expert Transformer layers that update atom representations using structure-local attention and a coordinate-based rotary positional encoding. Attention is bidirectional within observed structures, and new structures are generated atom by atom with an atom-type head followed by an EDM coordinate head. The model is continually pretrained on 3D structures and text, then fine-tuned separately on three small-molecule benchmarks introduced in the paper: conformational question answering, stereochemical comparison, and text-guided conformation editing. It is reported to achieve the best overall results against eight baselines in the benchmarks.

Critical strengths

Well-motivated injection of geometry throughout the language model. The paper identifies a clear limitation of existing 3D molecule-language models, which encode geometry only at the input. It addresses this with a simple design: expert Transformer layers with structure-local attention and coordinate-based rotary encoding, inserted after every fourth LLM layer. The layout ablation supports this choice: with expert capacity fixed, interleaving outperforms front- and back-stacking (Table 4) across all six question families (Table 19). The model also leads all three benchmarks by large margins and produces far more structurally valid edits than the baselines (Table 10). This makes the design a useful template for coupling continuous geometry with pretrained LLMs.

Critical weaknesses

Narrow evaluation that does not isolate the source of the gains. All results come from three benchmarks introduced in the paper (Table 6), which cover only small molecules and test generation only through conformation editing of the same molecule. Adding results on established benchmarks for tasks such as captioning, QA, or property-conditioned generation would help relate the findings to prior work. On the proposed benchmarks, AMMAS also differs from the baselines in three ways: continued pretraining on 3D structures (Table 5), the MET layers, and, for editing, an EDM coordinate head. The ablations, however, cover only MET placement and masking on conformational QA. Even the front-stacked variant exceeds the best baseline, so it remains unclear how much of the margin stems from interleaving, the paper’s central claim.

Questions for authors

Clarifications

  • How do the encoding in App. A.3, the canonicalization, and the coordinate head differ from those of InertialAR? Since the encoding is not rotation-invariant, how sensitive are the results to canonicalization (e.g., near-degenerate axes)?
  • In stereochemical comparison, A cannot attend to B. Was joint bidirectional attention across observed blocks, or cross-block MET attention in the common frame, considered, and are predictions consistent when A and B are swapped?
  • Could an input-stage geometry design (e.g., a 3D encoder with a projector), or Qwen3 with the same pretraining and EDM head, be evaluated to isolate MET’s contribution? Do the layout and mask ablations hold on the other two tasks?
  • Have the authors tried any established benchmark or de novo text-to-3D generation, which the architecture appears to support? Even preliminary results would be informative.
  • Baselines are fine-tuned with their original recipes and evaluated at the final checkpoint (App. C.1). How was it verified that they converged on the new benchmarks, and how do their training budgets compare with that of AMMAS?

Suggestions for future work

  • Extending the evaluation to proteins or materials, or scoping the title and claims to small molecules, would align the claims with the evidence.
Reviewer_zWLBOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

In this paper, the authors study language–3D structure modeling in atomistic science. They argue that existing methods encode geometry only at the input stage and then process structural and textual representations uniformly in the language backbone. To address this, they propose AMMAS, which interleaves pretrained LLM layers (SGT) with Modality Expert Transformer (MET) layers. MET layers apply structure-local attention and a three-axis rotary encoding of atomic coordinates, and structures are generated atom by atom using a type prediction head and a lightweight diffusion head for coordinates. The authors also introduce three small-molecule benchmarks: conformational question answering, stereochemical comparison, and text-guided conformation editing. Experiments verify the performance of AMMAS against LLM-based baselines.

Critical strengths
  • The paper proposes a unified framework that both understands and generates continuous 3D structures with language, without relying on serialized coordinate text or discrete structural codes.
  • The authors introduce three small-molecule benchmarks covering single-structure interpretation, cross-structure comparison, and conformation editing, and AMMAS outperforms baselines.
Critical weaknesses
  • I found something questionable about the design of the geometric module in the proposed AMMAS. The three-axis rotary encoding depends on the absolute orientation of the input, so the method has to rely on inertia-based canonicalization, which is unstable for near-symmetric molecules [1]. Besides, a single attention head cannot directly express inter-atomic distance, which is the key quantity in all three benchmarks, unlike equivariant or distance-based architectures [2]. Please refer to [3,4,5].
  • It seems that the comparisons in experimental part are not fair. AMMAS is further pretrained on large-scale 3D molecular data, while text-based baselines such as ChemDFM and Qwen3 never see coordinates before fine-tuning. More importantly, I hold concern for the reliability of the experimental results. In Table 10, the copy-source control returns the input unchanged, but I notice that only 98.51% and 98.57% of its outputs pass the connectivity and stereochemistry checks. Why does an unchanged input fail these checks? Does this mean that the structural screen itself misjudges some valid molecules? If so, how does this affect the validity-screened metrics of all models?
  • My major concern for this paper is the atom-by-atom generation design. Atoms cannot be adjusted once placed, which conflicts with the global constraints of molecular geometry such as ring closure and steric clashes. But, in Table 10, I found that only 71.06% of AMMAS outputs pass the structural check, much lower than the 91.45% of simply copying the input.

[1] Dym N, Lawrence H, Siegel J W. Equivariant frames and the impossibility of continuous canonicalization[J]. arXiv preprint arXiv:2402.16077, 2024.

[2] Satorras V G, Hoogeboom E, Welling M. E (n) equivariant graph neural networks[C]//International conference on machine learning. PMLR, 2021: 9323-9332.

[3] Liao Y L, Hoffman A J, Shen S C, et al. EquiformerV3: Scaling efficient, expressive, and general SE (3)-equivariant graph attention transformers[J]. arXiv preprint arXiv:2604.09130, 2026.

[4] Wu H, Wu L, Liu G, et al. SE3Set: Harnessing equivariant hypergraph neural networks for molecular representation learning[J]. arXiv preprint arXiv:2405.16511, 2024.

[5] Dumitrescu A, Korpela D, Heinonen M, et al. E (3)-equivariant models cannot learn chirality: Field-based molecular generation[C]//International Conference on Learning Representations. 2025, 2025: 30186-30220.

Questions for authors

Please check the weaknesses.

Reviewer_hCqsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AMMAS, a multimodal framework for jointly modeling 3D atomistic structures and natural language, aiming to bridge the gap between discrete linguistic representations and continuous geometric information. Its central design interleaves Shared Generalist Transformer (SGT) layers with Modality Expert Transformer (MET) layers, enabling repeated interactions between language and geometric representations throughout the network. The framework further incorporates role-aware attention masking and combines autoregressive atom-type prediction with diffusion-based coordinate generation to support both geometric understanding and language-guided 3D structure generation.

Experimental results show that AMMAS achieves the best overall performance among the evaluated baselines across three benchmarks. Architectural ablations further demonstrate the advantages of interleaved geometric expert layers for conformational question answering.

Critical strengths

The paper addresses a meaningful problem concerning the integration of 3D geometric information into language models. The overall architecture is well organized, with clearly defined roles for SGT and MET layers. The attention mechanisms, geometric positional encoding, and continuous coordinate generation process are described in sufficient detail, making the paper generally clear and easy to follow.

The experimental evaluation is reasonably comprehensive. The authors introduce three well-defined benchmarks and compare AMMAS against several language and molecular multimodal models. The inclusion of structural validity checks in the conformation editing task is particularly valuable, as it avoids relying solely on coordinate-based accuracy to assess generation quality. Furthermore, the MET placement ablations use matched initialization, pretraining protocols, and model capacity, providing convincing evidence for the effectiveness of interleaved expert layers in conformational question answering.

Critical weaknesses

The contribution of geometric continued pretraining is not sufficiently disentangled from that of the proposed architecture. Since 65% of the effective token budget is allocated to 3D molecular data, comparisons with baselines without comparable geometric pretraining make it difficult to attribute the reported improvements specifically to the SGT–MET architecture.

The architectural ablations are limited to conformational question answering. It remains unclear whether the benefits of interleaved geometric processing extend to continuous coordinate generation, which is another central contribution of the paper. Direct ablations on the conformation editing task would strengthen this claim.

The methodological novelty also needs clearer justification. Several geometric components, including positional encoding, coordinate canonicalization, and EDM-based coordinate prediction, are explicitly adopted from InertialAR (ICML 2026). While the interleaved SGT–MET design provides a potentially meaningful contribution, its advantages over existing multimodal fusion architectures are not sufficiently established.

Questions for authors
  1. Could the authors provide a control using comparable geometric pretraining data and computational budgets to better distinguish the contributions of continued pretraining from those of the proposed architecture? If such an experiment is infeasible, could the authors clarify the extent to which the cross-baseline improvements can be attributed to the architecture itself?
  2. The MET placement ablations in Table 4 are limited to conformational question answering. Could the authors compare interleaved, front-stacked, and back-stacked configurations on the text-guided conformation editing benchmark to establish whether the architectural advantage extends to continuous-coordinate generation?
  3. The contribution of bidirectional observed-block attention is evaluated only on conformational question answering. Could the authors provide corresponding ablations on conformation editing or stereochemical comparison to clarify whether this mechanism is similarly beneficial for other tasks?
  4. AMMAS explicitly builds upon InertialAR for several geometric modeling components, including positional encoding, coordinate canonicalization, and diffusion-based coordinate prediction. Meanwhile, modality-specific expert architectures have already been explored in CogVLM and Mixture-of-Transformers. Beyond combining these established techniques, what specific advantage does the proposed interleaved SGT–MET interaction mechanism provide that existing designs cannot readily achieve? Could the authors support this distinction through additional architectural comparisons or more targeted mechanism analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.