acceptodds
Withdrawn from ICLR 2027

Chem4DLLM: 4D Multimodal LLMs for Chemical Dynamics Captioning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing chemical understanding tasks primarily rely on static molecular representations, limiting their ability to model inherently dynamic phenomena such as bond breaking or conformational changes, which are essential for a chemist to understand chemical reactions. To address this gap, we introduce Chemical Dynamics Understanding (ChemDU), a new task that translates 4D molecular trajectories into interpretable natural-language explanations. ChemDU focuses on fundamental dynamic scenarios, including simple molecular motion, gas-phase reactions and catalytic reactions, and requires models to reason about key events along molecular trajectories, such as bond formation and dissociation, and to generate coherent, mechanistically grounded narratives. To benchmark this capability, we construct Chem4DBench, the first dataset pairing 4D molecular trajectories with explanations across these settings. We further develop a suite of 4D molecular language models, including 4D Text-based, 4D-MolT5 and Chem4DLLM. Notably, Chem4DLLM is a trajectory-aware model that integrates an equivariant graph encoder with a pretrained large language model to capture molecular geometry and temporal dynamics. This model suite enables a systematic evaluation of how different 4D representations and architectures support dynamic chemical understanding. We hope that ChemDU, together with Chem4DBench and our 4D model suite, will stimulate further research in dynamic chemical understanding and multimodal scientific reasoning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
4D molecular dynamics, multimodal LLMs, chemical dynamics, molecular trajectories

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rB1LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose ChemDU a task that translates 4D molecular trajectories into grounded text, Chem4DBench a benchmark for 4D molecular understanding and Chem4DLLM model. Chem4DLLM model encodes each frame using 3D-equivariant MLIP model UMA, which are projected together with the Cartesian Coordinates x,y,z, atomic number and PBC into LLM (Qwen 3-8B ) one token per atom per frame. The authors compared the proposed method againt 4D model variant of 3D-MolT5 they designed and 3D-MolT5 and show that Chem4DLLM has better results in most of the cases.

Critical strengths

None

Critical weaknesses
  • The comparison is not controlled. 3D-MolT5 and 4dMolT5 use a different LLM backbone (T5), smaller than the one that Chem4DLLM uses (Qwen3-8B). Chem4DLLM uses UMA for the structure encoder and one token per frame, compared to E3FP fingerprints and flattened fingerprint sequence used by 3D-MolT5.

  • The time dimension is handled implicitly by LLM, and inputs aren't rotation safe. The authors use UMA to encode each frame separately, so they became independent sets of 3D features. The 4D part of time and change is not computed by any structural component so the LLM has to learn by itself to match atoms across frames. Although UMA is equivariant by design, the authors add raw x,y,z coordinates to each atom feature, which will change if the molecule rotates even if chemistry is identical. They keep l=1 features (vectors) which will change direction with rotation. Both of them are fed to LLM without rotation information, so the same molecule in two different orientations will be seen as two different inputs.

Questions for authors
  1. The Authors use UMA to generate the 6000 extra catalysis data and use the same UMA model to encode the input features. This will overstate how well the model will do when using DFT or experimental data. Why did authors choose the same model for two different tasks?
  2. For all the datasets what is the UMA barrier = E(Transition State) - E(reactant)? How much is the difference (error) between UMA barrier and DFT barrier for all datasets? What is the difference between UMA's error and LLM's error for all the datasets?
  3. How will the performance of 3D-MolT5 change if the same LLM backbone is used under a controlled comparison?

Suggestions

  • What UMA model and tasks are used in this work?
  • "addrss" line 207 page 4
  • Table 1 is not needed
  • Figure 2 is difficult to read due to the colors used
  • Consistency on the paragraph titles, some have "." some don't
  • Evaluation paragraph would fit better in the section 5
Reviewer_cz63OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how LLMs can interpret molecular trajectories and describe chemical dynamics. The authors approach this problem from both the data and model sides. On the data side, the authors argue that existing datasets focus largely on static properties or non-periodic molecules, leaving limited coverage of dynamics of moleecules and consideration of periodic systems such as crystals. To address limitation, the authors construct Chem4DBench, which covers the molecular dynamics across reaction product prediction, catalytic reactions, and molecular motion.

On the model side, the main limitation is the difficulty of encoding long molecular trajectories and reasoning about changes and dependencies over time. Chem4DLLM, the proposed model, leverages UMA atom features with Cartesian coordinates, projects them into node-token embeddings, and passes the frame sequence to Qwen3-8B. It further reduces the input length by keeping atom-level tokens only when the maximum atomic displacement from the last retained frame exceeds a threshold. The other frames retain only their boundary tokens.

Experimentally, Chem4DLLM generally performs better than the evaluated baselines on the trajectory-related tasks. The ablation study also shows that including explicit coordinates improves motion classification.

Critical strengths
  • Understanding molecular trajectories is a timely and important problem. A static structure gives only a partial view of a molecular system, and learning to interpret how it changes could make molecular LLMs more useful for studying chemical processes.
  • Using UMA as the geometric encoder makes sense. Its invariant atom features capture the local chemical environment, while the added coordinates preserve information about molecular motion.
  • I find the inclusion of catalytic reactions particularly interesting. Covering periodic catalyst–adsorbate systems as well as gas-phase reactions gives the benchmark a broader scope than isolated-molecule motion alone.
Critical weaknesses
  • My major concern is the gap between the goal of understanding molecular dynamics and the tasks considered in the benchmark. The constructed dataset, Chem4DBench, focuses on specific prediction tasks such as SMILES prediction. These prediction tasks limit the scope of the constructed benchmark, especially for the broader knowledge and reasoning needed to understand molecular mechanisms and dynamics. The example of a binding-induced conformational change in the introduction suggests a richer goal, but the constructed dataset superficially reflects this goal. It would be better to discuss about the definition of the knowledge and reasoning abilities that a molecular dynamics LLM should learn, such as the interactions driving structural changes, and explain how these abilities can be diluted in the design of the benchmark.
  • The connection between the proposed model design and the challenges in the introduction is relatively weak. The encoder and projector address how to model molecular geometry into an LLM, and frame selection addresses input length. However, it is less clear how the model architecture design addresses long-range dependencies or causal relationships. These abilities may depend not only on the architecture but also on the training examples and the architecture. To clarify this point, I suggest to show whether the model can use an earlier event to explain a later one. Examples involving multiple separated events would help support this part of the motivation.
  • The term "coarse-to-fine trajectory tokenization" seems somewhat misleading. The proposed method keeps atomic representations for selected frames based on a movement threshold through frames, and a progressive coarsening/fine-graining steps are not explained in this process. "Frame selection based on atomic displacement" would describe it more directly.
  • It is unclear how well the learned capabilities transfer to new tasks. The test sets evaluate new molecular examples, but they share the same prediction tasks. Therefore, it is unclear whether the model can perform tasks outside its fine-tuning tasks. Evaluation on new tasks, such as explaining why a molecular change occurs or how it influences subsequent events, would give stronger evidence of transferable understanding.
  • The ablation study is limited to some specific design choices, such as whether to include explicit coordinates or equivariant UMA features. For the data contribution, it would be useful to remove one category, such as catalytic reactions, and measure the effect on both the removed task and the other tasks. On the model side, all variants in Table 5 retain UMA features. I suggest to see an ablation that removes UMA features entirely, as well as a comparison with all frames retained. Reporting performance and computational costs across different displacement thresholds would also clarify the benefit of frame selection.
  • The experimental setup for implementing baselines is unclear. Chem4DLLM uses Qwen3-8B, while 4D-MolT5 is T5-based. The exact checkpoint or size of 4D-MolT5, or the backbones used by the text-based baselines are not clearly explained. It is also unclear whether all baselines receive comparable task-specific fine-tuning. A comparison using the same language backbone and comparable training settings would help establish how much the proposed trajectory representation contributes.
Questions for authors
  • Have the authors evaluated Chem4DLLM on tasks excluded from fine-tuning?
  • What happens to performance when one data category is removed from training?
  • How much do UMA features add over coordinates alone?
  • Compared to keeping all frames, how does frame selection affect accuracy, token counts, runtime, and memory use? How sensitive are these results to the displacement threshold?
Reviewer_2QGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce Chem4DLLM to address a key limitation of existing molecule-language models, which typically more of a single static molecular snapshot despite chemistry being inherently dynamic, involving atomic motion, bond breaking, and surface reactions. They propose ChemDU and Chem4DBench, a benchmark covering both gas-phase reaction product prediction and catalytic surface reactions. Chem4DLLM encodes trajectory frames using the pretrained equivariant UMA model, projects atom-level representations into the Qwen3-8B embedding space and fine-tunes the LLMs. The method substantially outperforms 3D, text-serialized, and T5-based baselines on in-distribution datasets and achieves good performance on catalytic reaction tasks. However, the performance advantage largely vanishes on out-of-distribution splits, the paper provides limited ablation analysis, and the generated outputs are primarily structured property predictions rather than the richer narrative explanations suggested by the paper's motivation.

Critical strengths

The paper's most significant contribution is establishing a clear vision for 4D molecular understanding, extending the progression from 1D molecular strings to 2D, 3D, and dynamic molecular trajectories. This framing provides the community with a useful conceptual framework and vocabulary through the introduction of ChemDU. In addition, the benchmark goes beyond isolated molecules by incorporating periodic catalytic surface systems which are definitely important but less covered, broadening the scope of molecular-language modeling to more realistic chemistry and broader fields application. The evaluation is also thoughtfully designed generally.

Critical weaknesses
  1. Necessity of the LLM is not established. The benchmark’s targets e.g. product SMILES, barrier, enthalpy, reaction type, TS step... are structured outputs that are wrapped in a fixed natural-language template, and the evaluation is entirely based on SMILES similarity and regression error. Such quantities can be obtained directly from machine-learned interatomic potentials including UMA, which is the paper’s own encoder or from a standard GNN with a task-specific head. The paper provides no such baselines, no mean-predictor reference, and no evaluation of free-form explanation quality, so it is unclear whether the LLM adds capability or simply acts as a costly output decoder over pretrained UMA features. The motivating claims interpretable narratives, causal reasoning, agentic discovery are not tested.

  2. A static 3D model fed reactant and product geometries like a UMA-based energy calculation on those frames could predict barrier and enthalpy without any 4D machinery. e.g. Enthalpy is just the energy difference between product and reactant. The paper never compares against such a task appropriate static baseline, only against static models that lack the right inputs.

  3. A convincing 4D benchmark should include tasks that directly require and benefit from that information. However, most tasks in this benchmark do not clearly demonstrate that the 4D trajectory information is necessary, as many appear solvable from static structural representations alone.

Questions for authors

The reaction product task only uses two frames reactant and transition state. Does that really count as the claimed 4D trajectory, and would results change with more frames? Exact-match accuracy drops from about 0.53 in-distribution to 0.05-0.12 on the OOD splits, and on OOD-Reactants it is worse than the T5-based baselines. Does this mean the model is mostly pattern matching on scaffolds, any possible reasons for that? Is there a specific reason the authors chose UMA over other models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.