- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PolyWorld, a multimodal scientific reasoning benchmark for evolving polymer systems, aiming to address the limitations of existing materials science benchmarks that primarily focus on static structural recognition, material property prediction, and knowledge-based question answering, with limited evaluation of physical evolution and intervention-based decision-making. Based on structural information, physical measurements, and intervention outcomes, the authors construct 4,178 multimodal questions spanning four categories: perception, reasoning, decision-making, and planning. The benchmark is evaluated using 13 representative multimodal large language models.
Critical strengths
- Current benchmarks in materials science primarily focus on static structures, material properties, or scientific knowledge. In contrast, actual scientific research often requires predicting how a system evolves under external interventions and selecting appropriate actions based on those predictions. This paper extends the scope of evaluation to include dynamic physical evolution and intervention-based decision-making, offering significant research value.
- The benchmark construction is systematic and grounded in physical simulations. The authors use molecular dynamics simulations to generate physical evolution trajectories under different intervention conditions, rather than relying solely on manually designed or LLM-generated questions. In addition, deriving ground-truth answers from simulation outcomes and predefined rules helps ensure the objectivity and reproducibility of the evaluation.
Critical weaknesses
- It remains unclear whether the benchmark adequately evaluates dynamic physical reasoning and planning capabilities. Although the paper aims to assess models' ability to reason about physical evolution, make intervention decisions, and perform multi-step planning, most tasks are formulated as offline multiple-choice questions. In particular, the Planning tasks mainly require models to select from predefined action sequences, without actually interacting with the environment, receiving feedback, or adjusting subsequent actions. Therefore, it is unclear to what extent these tasks reflect the dynamic planning capabilities required by real-world scientific agents. Furthermore, some Reasoning tasks mainly involve comparing sequential images, curve trends, or predefined physical quantities, which may be solved through general visual understanding or chart interpretation without necessarily requiring an understanding of the underlying physical mechanisms.
- The comparison between human experts and models is not sufficiently fair. Table 2 directly compares the 71.25% accuracy achieved by human experts on a subset of 80 questions with the 44.28% accuracy of the best-performing model on the full set of 4,178 questions, highlighting a performance gap of approximately 26.97 percentage points. However, according to Appendix A.4, the best-performing model achieves 58.75% accuracy on the same 80-question subset, corresponding to an actual gap of only 12.5 percentage points under matched evaluation conditions. The authors should report comparisons on the same evaluation subset in the main results and revise the corresponding conclusions accordingly.
Questions for authors
- Table 2 shows that GPT-5.6 Sol achieves only 24.60% and 23.85% accuracy on Trade-off Decision and Adaptive Planning, respectively, both falling below the corresponding random-guessing baselines. How do the authors explain this observation? It would be helpful to report random-guessing baselines and the distribution of predicted answer choices for each subtask to better understand the reasons behind these results.
- According to Appendix A.1, the Planning tasks are primarily constructed using two-stage interventions, with only three candidate actions at each stage. However, the paper uses these results to discuss the limitations of current models in long-horizon planning. To what extent can two-step planning tasks support such conclusions? Could the authors include experiments with longer action sequences to evaluate model performance across different planning horizons?
4 comments
Sincerely looking forward to the further discussions
Dear reviewer,
We are wondering if our response and revision have resolved your concerns. If our response has addressed your concerns, we would highly appreciate it if you could re-evaluate our work and consider raising the score.
If you have any additional questions or suggestions, we would be happy to have further discussions.
Best regards,
The Authors
Thank you for the detailed response. It has addressed my questions and concerns. I will consider raising my score.
Reply to Reviewer fd8C
Thank you for reconsidering our work and raising your score! We truly appreciate your helpful feedback, which has helped us improve our paper.
Reply to Reviewer fd8C
We thank Reviewer fd8C for recognizing the value of extending materials-science evaluation to dynamic physical evolution and intervention-based decision-making. We respond to the questions below.
W1:
We thank the reviewer for raising this important point. We clarify the scope and design of PolyWorld below.
Offline benchmark scope. We agree that PolyWorld is offline and short-horizon, but offline evaluation with fixed tasks and rule-based or ground-truth verification is a common practice in influential agent benchmarks[1,2], which have been widely used to evaluate multi-step reasoning, tool use, and action selection. PolyWorld similarly tests a verifiable capability, reasoning about intervention consequences from observed physical evolution, with labels reproducibly derived from simulation outcomes and predefined rules. We will clarify this positioning and include interactive, longer-horizon evaluation in our future work.
Chart-reading concern. To test whether performance depends on physical understanding rather than general chart interpretation, we add a current-state-only control on a fixed stratified subset of 200 Decision and Planning questions (50 each for D1/D2/P1/P2) using GPT-5.6 Sol. In this control, the current conformation, conditions, and candidate options are unchanged, but the history chart are hidden.
Removing history reduces accuracy substantially on D1 and P1 (16 and 14 pp), indicating that models use temporal evolution information rather than static chart cues alone. We will report this analysis in the revision and temper any claim not supported by the control.
W2
We thank the reviewer for pointing this out. Due to the high cost of expert annotation, human evaluation was conducted on a subset rather than the full benchmark, which is a common practice in scientific benchmarks [3–5]. We agree that comparisons should be made on the same subset. In the revision, we will report the matched 80-question comparison as the primary human–model result and report full-benchmark model accuracy separately. We will revise the corresponding conclusions and add further analysis accordingly.
Q1
We analyze GPT-5.6 Sol's pressure-choice predictions on D2 and P2 and report the distributions below. As shown, the model exhibits a strong systematic preference for 1 atm, especially on P2, which likely contributes to its below-random accuracy. The corresponding random-guessing baselines are 33.1% for D2 and 33.3% for P2, since not all items have four options. We will add further analysis in the revision.
D2 pressure-choice distribution (226 incorrect answers)
P2 pressure-choice distribution (202 incorrect answers)
Q2
We will revise the corresponding claims to explicitly scope our conclusions to multi-step planning under controlled interventions, rather than implying long-horizon planning capability. We will also discuss longer action sequences and long-horizon planning as future work.
We hope the above response can further address your concern. If you have any further questions, we are pleased to discuss them.
[1] Mialon, Grégoire, et al. "Gaia: a benchmark for general ai assistants." International Conference on Learning Representations. Vol. 2024. 2024.
[2] Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." International Conference on Learning Representations. Vol. 2024. 2024.
[3] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., ... & Chen, W. (2024). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9556-9567).
[4] Mirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., ... & Jablonka, K. M. (2025). A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, 17(7), 1027-1034.
[5] Lai, Z., Zheng, Y., Cai, Z., Lyu, H., Yang, J., Liang, H., ... & Wang, B. (2025). Can multimodal llms see materials clearly? a multimodal benchmark on materials characterization. arXiv preprint arXiv:2509.09307.