acceptodds
Under review as a conference paper at ICLR 2027

MultiPhaseWorld: A Multimodal Benchmark for Structural Evolution in Multiphase Systems

Abstract

Large language models (LLMs) and multimodal large language models (MLLMs) are increasingly applied to scientific problems. However, existing scientific benchmarks primarily evaluate factual knowledge and static structure understanding, leaving evolution reasoning in diverse chemical environments underexplored. We introduce , a multimodal benchmark for evaluating scientific reasoning over structural evolution in multiphase systems. Multiphase systems involve complex interactions across phases and diverse chemical environments, making structural evolution challenging to understand. At the same time, this complexity provides a rich setting for evaluating the reasoning capabilities of LLMs and MLLMs across multiple scales, including adsorbate, surface, interface, layer, and bulk environment. MultiPhaseWorld contains 11,075 text and multimodal questions across 28 tasks and 14 capability dimensions, organized into four progressive levels: , , , and . The benchmark is built from 336 solid-solid and 400 solid-gas systems. Controlled structural perturbations, structural optimization, and molecular dynamics simulations are used to generate structural responses, evolved structures, and evolution trajectories. The strongest models achieve only 58% average accuracy on text tasks and 51% on multimodal tasks, remaining more than 20 percentage points below the expert performance. Fine-grained results further reveal limitations in integrating physicochemical knowledge, reasoning about evolution relationships, and resolving subtle structural changes. Overall, current models show partial understanding of chemical information, but remain limited in linking explicit perturbations to structural responses, tracking state evolution, and predicting perturbation effects.

Primary area:
datasets and benchmarks
Keywords:
Structural Evolution Reasoning, Multimodal Scientific Benchmark, Foundation Model Evaluation, Multiphase Materials

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RPRuOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces MultiPhaseWorld, a simulation-grounded benchmark evaluating whether LLMs and multimodal LLMs can connect structural perturbations to their consequences and reason about subsequent evolution in multiphase chemical systems, beyond static structure understanding. It contains 11,075 text and multimodal questions across 28 task types and 14 capability dimensions, organized into four levels: multiphase structure perception, perturbation response reasoning, structural evolution reasoning, and structural prediction. Built from 336 solid–solid heterostructures and 400 solid–gas surface–adsorbate systems, it uses controlled atomic, molecular, surface, layer, and environmental perturbations, followed by structural optimization and molecular dynamics, to generate evaluation data.

Critical strengths

This benchmark seems solid and is beneficial to LLM and Agent research community.

Critical weaknesses

None

Questions for authors

Can you explain why this is important and how it defers from previous benchmarks like AtomWorld [1] and EvoMD-LLM [2]?

[1] Taoyuze Lv, Alexander Chen, Fengyu Xie, Chu Wu, Jeffrey Meng, Dongzhan Zhou, Yingheng Wang, Bram Hoex, Zhicheng Zhong, and Tong Xie. AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials. arXiv preprint arXiv:2510.04704, 2026. [2] Zhichen Tang, Zhengzheng Dang, Yulin Chen, Jixin Wu, Haiwen Li, and Yanming Wang. EvoMD-LLM: Learning the Language of Species Evolution in Reactive Molecular Dynamics. arXiv preprint arXiv:2605.29394, 2026.

Reviewer_U3LdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper the author put together MultiPhaseWorld, a benchmark designed to test whether language models can understand how material structures respond and change after a perturbation. The benchmark includes a large set of text based or image-related questions covering structure recognition, predicting responses to changes and reasoning about structural evolution. The results show that even the strongest current models perform well below human experts, especially on tasks that require reasoning about stability and structural changes over time.

Critical strengths

The novelty of this work lies in combining structure understanding, perturbation reasoning, and evolution prediction into a single benchmark that supports both text and image inputs. The authors argue that this broader evaluation setting goes beyond existing benchmarks, which typically focus on only one or two of these aspects with a reasonable design. By integrating these tasks into a framework, the benchmark can be a more comprehensive assessment of a model’s ability to reason about structural changes. In addition, the benchmark itself and the automated pipeline for generating benchmark questions could be useful for developing future benchmarks in related domains.

Critical weaknesses

There are several limitations. The benchmark is built from a relatively small set of initial structures, and the labels rely on simulations using a machine-learning potential rather than higher-accuracy quantum chemistry calculations like DFT, which are generally considered the gold standard for evaluating structure stability and energetics. In addition, I did not find a detailed analysis of liked random-guessing baselines or task difficulty. As a result, it is difficult to assess whether poor model performance reflects genuine challenges in physical reasoning or simply the inherent difficulty and design of the benchmark itself. Additional analysis comparing model performance against simple baselines and examining the underlying physical factors that drive task difficulty would help strengthen the paper's conclusions.

Questions for authors

How is the random baseline computed, and how many answer options do each task have? Is the correct-answer position balanced, and are classes balanced for the binary tasks?

For several tasks the models are at chance or worse, such as T6 0.52 vs. 0.50, T8 0.20 vs. 0.32. Is below-chance performance across all models caused by answer-position bias, distractor construction or a label artifact?

1 comment

Thank you for the care you have taken in reviewing our work. If you have a chance before the discussion closes in about two days, we would greatly appreciate hearing whether any concerns remain.

Reviewer_JJUdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MultiPhaseWorld, a multimodal benchmark for evaluating how well LLMs understand and reason about structural changes in solid–solid and solid–gas systems following physical and chemical perturbations. Using controlled perturbations, structural optimization, and molecular dynamics simulations, the paper constructs 11,075 questions spanning 28 tasks across four capability levels: structure perception, perturbation response, structural evolution, and structural prediction. Under the reported evaluation settings, the best results among 13 models reach 58% average accuracy on text tasks and 51% on multimodal tasks.

Critical strengths
  1. The paper addresses a meaningful gap in existing scientific benchmarks by shifting the evaluation focus from static structure understanding and manipulation to reasoning about structural responses following explicit perturbations. Its four-level task taxonomy provides a systematic framework for evaluating these related but distinct capabilities.

  2. The authors develop a comprehensive pipeline covering 736 solid–solid and solid–gas systems, five categories of structural perturbations, and 11,075 text and multimodal questions. Reference answers are derived programmatically from structural information, simulation outputs, and explicit labeling rules rather than LLM-generated annotations. The detailed documentation of perturbation procedures, simulation settings, and filtering criteria makes the construction process relatively transparent and auditable.

Critical weaknesses
  1. The benchmark does not measure the claimed dynamic scientific reasoning capabilities. A huge portion of the structural evolution tasks rely on BFGS optimization trajectories, where the ordering of states reflects numerical optimization steps rather than physical time evolution. Though these trajectories are useful for studying structural relaxation, it is unclear whether recovering their order genuinely measures physical evolution reasoning. Moreover, the reconstruction metric includes direct atomic substitutions and deletions, potentially conflating the imposed perturbations with the subsequent physical responses. Without appropriate controls or simple heuristic baselines, the benchmark does not sufficiently establish that its tasks isolate the intended reasoning capabilities.

  2. The physical reliability of the simulation-derived ground truth is not validated. The benchmark relies heavily on labels derived from a single interatomic potential (MACE-MP-0) without systematic validation against higher-fidelity calculations or alternative simulation settings. In particular, the solid–solid systems are based on van der Waals heterostructures, but the simulations are performed without dispersion corrections, which may affect interlayer interactions, structural relaxation, and the resulting stability or reconstruction labels. Although the authors apply geometric and convergence-based filtering, these checks do not establish the physical accuracy or robustness of the reference answers.

Questions for authors
  1. For T5/MT3, the task objective is to identify the most interface-concentrated reconstruction. However, Appendix E defines interface concentration using D_int = R_int / R_edit, while the reference answers are generated by ranking candidates according to R_int. Could the authors clarify which metric is actually used to determine the correct answers? How frequently would the answers differ if candidates were ranked by D_int instead?

  2. The paper reports that the strongest models underperform human experts by more than 20 percentage points. However, the expert evaluation uses a 70-question subset, while the reported model scores are computed over the full benchmark with different task distributions. Could the authors report model performance on the exact same questions to establish a directly comparable human–model performance gap?

  3. The paper reports lower aggregate accuracy on multimodal tasks and attributes this partly to difficulties in extracting structural information from images. However, the text and multimodal benchmarks contain different task sets. Could the authors provide a paired comparison using identical questions and answer options across the two modalities?

4 comments

We sincerely appreciate your time and careful consideration. As the discussion draws to a close in about two days, we hope our responses have been helpful and would welcome any additional comments you might wish to share.

Thank you for the additional experiments and clarifications. The matched human–model comparison addresses Q2, and the DFT/D3 validation provides encouraging evidence regarding ground-truth reliability.

However, my main concern (W1) remains unresolved. Outperforming simple geometric heuristics does not establish that recovering BFGS optimization trajectories measures physical evolution reasoning rather than numerical relaxation patterns.

For W2, I would appreciate clarification on the DFT validation protocol, particularly the sampling strategy, whether structural relaxation was performed, and how label agreement was calculated.

Regarding Q1, the claimed task objective should be reconciled with Table 5, especially given the reported 23% disagreement between the two metrics. For Q3, the matched results suggest a much smaller modality gap than originally reported, warranting a more cautious interpretation.

Finally, the paper involves strongly chemistry-related content, which is outside my area of expertise, I can only provide my opinion about the paper but cannot increase my confidence rating or recommend acceptance/rejection. If the AC finds that some of my concerns are off the mark, please feel free to set them aside.

Thank you for your further comments and for acknowledging our additional experiments and clarifications. Given the limited time remaining in the discussion period, we would like to provide brief clarifications on the following points.

W1. Physical interpretation of BFGS trajectories.

We apologize for not sufficiently clarifying this aspect previously. We acknowledge that BFGS trajectories do not fully represent real-time physical evolution. However, the four sampled frames exhibit a consistent decreasing energy trend, following the fundamental energetic principle that structures tend to evolve from higher- to lower-energy states. Thus, although they are not actual dynamical pathways, they represent physically plausible structural relaxation processes and retain meaningful physical information about perturbation-induced structural responses.

W2. DFT validation protocol.

We employed balanced stratified sampling across atom-count ranges and perturbation types. No additional structural relaxation was performed. Instead, static DFT calculations were conducted on the sampled BFGS structures under consistent computational settings. We compared the relative energy trends from DFT and MACE-MP-0 to assess their agreement.

Label agreement was calculated as the fraction of trajectories for which the DFT-derived option was included in the accepted reference answer set:

$ A_{\mathrm{agreement}}=\frac{1}{N}\sum_{t=1}^{N} \mathbb{I}\left[\hat{y}^{\mathrm{DFT}}_t\in Y^{\mathrm{ref}}_t\right].$

Q1. Reconstruction metrics.

We have removed the $D_{\mathrm{int}}\) formulation from the revised manuscript and updated the corresponding task descriptions and tables. (R_{\mathrm{int}}\) measures the absolute extent of interface reconstruction, whereas ($D_{\mathrm{int}}measuresitsrelativeconcentrationattheinterface.Sinceourtaskconcernsabsolutereconstructionintensity,measures its relative concentration at the interface. Since our task concerns absolute reconstruction intensity,R_{\mathrm{int}}$$ is the appropriate metric.

Q3. Text–multimodal comparison.

The paired evaluation shows a substantially smaller modality gap than the original aggregate results. We will revise the corresponding discussion in the next manuscript version to provide a more cautious interpretation.

We will also incorporate these clarifications into the manuscript and upload the updated version shortly.

We sincerely appreciate your constructive feedback and understanding, which have helped improve our work.

Thank you for your valuable feedback and constructive suggestions. In response, we have made the following revisions to the manuscript.

For W1, we have further clarified the energy-based frame selection criterion, explicitly stating that all four sampled frames satisfy a monotonically non-increasing energy trend (E1≥E2≥E3≥E4E_1 \geq E_2 \geq E_3 \geq E_4).

For W2, we have updated the DFT computational details in Appendix D.

For Q1, we have checked the description in Table 5 to ensure consistency with the stated task objective and updated Appendix E to report the 23% disagreement between the two metrics.

For Q3, we have revised the relevant statements in the Introduction and Main Results sections, with all modifications highlighted in red.

The updated manuscript has now been uploaded.

We sincerely appreciate your time and thoughtful feedback, which have helped us improve the clarity and rigor of our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thank you for your valuable suggestions. We look forward to further discussion!

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We have addressed the reviewers' concerns through the following clarifications and improvements in the new version of our paper, with all modifications highlighted in red:

  1. Physical Reliability. Added DFT validation and D3 sensitivity experiments to assess the physical reliability and robustness of the reference labels (Appendix D).

  2. Heuristic Baselines. Added heuristic baselines to evaluate whether the tasks require complex reasoning beyond simple geometric comparisons (Appendix F).

  3. Human–LLM Comparison. Added matched evaluations of human experts and LLMs on identical questions. The results confirm the performance gap between them (Table 11).

  4. Paired Text and Multimodal Evaluation. Conducted paired comparisons using identical questions and answer options. The results are consistent with the trends reported in the paper (Table 12).

  5. Random Baselines and Answer Distributions. Clarified the calculation of random-guessing accuracy (Appendix F) and provided the number of options for all tasks, together with analyses of answer-position and class distributions (Appendix A.3).

  6. Performance Analysis and Benchmark Positioning. Analyzed near- or below-chance performance on T6/T8 and further clarified the distinctions between MultiPhaseWorld, AtomWorld, and EvoMD-LLM.

  7. Interface Reconstruction Metric. Corrected the outdated description in Appendix E and clarified the corresponding task description in Table 5, ensuring consistency among the task objective, reference answers, and evaluation procedure.

For detailed responses, please kindly refer to our individual rebuttal responses in the discussion section.

We sincerely hope that these revisions and additional experiments have addressed the concerns raised. We would greatly appreciate any further feedback and look forward to continuing the discussion with you.

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your time and effort throughout the review process.

We appreciate the reviewers’ shared recognition of the informativeness of our work and their positive assessment of its systematic task taxonomy, transparent benchmark construction, and the automated pipeline’s potential to support future benchmark development.

Based on the reviewers’ valuable feedback, we have made several improvements, including DFT validation, D3 sensitivity experiments, heuristic baselines, matched evaluations, and clearer descriptions of the evaluation metrics and benchmark positioning. These revisions strengthen the paper’s soundness and presentation and directly address the reviewers’ questions and concerns. All changes are highlighted in red in the latest version of the paper, with detailed responses provided in the discussion.

As the discussion period draws to a close, we would be grateful for any further feedback you may wish to share, should your schedule permit. Thank you again for your consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.