acceptodds
Under review as a conference paper at ICLR 2027

PolyWorld: Benchmarking Multimodal Reasoning over Evolving Polymers

Abstract

Multimodal large language models (MLLMs) have shown strong capabilities in understanding images and solving scientific problems, but most existing benchmarks focus on static observations, property prediction, or knowledge-based reasoning. They rarely test whether models can understand how a physical system changes after an intervention and use this understanding to guide subsequent actions. We introduce PolyWorld, a multimodal benchmark for physical reasoning over evolving polymers. PolyWorld contains 5,530 diverse 3D polymer structures built from 2,333 unique repeat units. We use controlled interventions and molecular dynamics simulations to create 4,093 multimodal questions spanning four capabilities: perception, reasoning, decision, and planning. Our evaluation reveals a substantial gap between current MLLMs and human experts across these tasks. These results show that understanding and acting on evolving physical systems remains challenging for multimodal models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Multimodal Large Language Models, Scientific Reasoning, Polymer Science, Multimodal Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fd8COpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyWorld, a multimodal scientific reasoning benchmark for evolving polymer systems, aiming to address the limitations of existing materials science benchmarks that primarily focus on static structural recognition, material property prediction, and knowledge-based question answering, with limited evaluation of physical evolution and intervention-based decision-making. Based on structural information, physical measurements, and intervention outcomes, the authors construct 4,178 multimodal questions spanning four categories: perception, reasoning, decision-making, and planning. The benchmark is evaluated using 13 representative multimodal large language models.

Critical strengths
  1. Current benchmarks in materials science primarily focus on static structures, material properties, or scientific knowledge. In contrast, actual scientific research often requires predicting how a system evolves under external interventions and selecting appropriate actions based on those predictions. This paper extends the scope of evaluation to include dynamic physical evolution and intervention-based decision-making, offering significant research value.
  2. The benchmark construction is systematic and grounded in physical simulations. The authors use molecular dynamics simulations to generate physical evolution trajectories under different intervention conditions, rather than relying solely on manually designed or LLM-generated questions. In addition, deriving ground-truth answers from simulation outcomes and predefined rules helps ensure the objectivity and reproducibility of the evaluation.
Critical weaknesses
  1. It remains unclear whether the benchmark adequately evaluates dynamic physical reasoning and planning capabilities. Although the paper aims to assess models' ability to reason about physical evolution, make intervention decisions, and perform multi-step planning, most tasks are formulated as offline multiple-choice questions. In particular, the Planning tasks mainly require models to select from predefined action sequences, without actually interacting with the environment, receiving feedback, or adjusting subsequent actions. Therefore, it is unclear to what extent these tasks reflect the dynamic planning capabilities required by real-world scientific agents. Furthermore, some Reasoning tasks mainly involve comparing sequential images, curve trends, or predefined physical quantities, which may be solved through general visual understanding or chart interpretation without necessarily requiring an understanding of the underlying physical mechanisms.
  2. The comparison between human experts and models is not sufficiently fair. Table 2 directly compares the 71.25% accuracy achieved by human experts on a subset of 80 questions with the 44.28% accuracy of the best-performing model on the full set of 4,178 questions, highlighting a performance gap of approximately 26.97 percentage points. However, according to Appendix A.4, the best-performing model achieves 58.75% accuracy on the same 80-question subset, corresponding to an actual gap of only 12.5 percentage points under matched evaluation conditions. The authors should report comparisons on the same evaluation subset in the main results and revise the corresponding conclusions accordingly.
Questions for authors
  1. Table 2 shows that GPT-5.6 Sol achieves only 24.60% and 23.85% accuracy on Trade-off Decision and Adaptive Planning, respectively, both falling below the corresponding random-guessing baselines. How do the authors explain this observation? It would be helpful to report random-guessing baselines and the distribution of predicted answer choices for each subtask to better understand the reasons behind these results.
  2. According to Appendix A.1, the Planning tasks are primarily constructed using two-stage interventions, with only three candidate actions at each stage. However, the paper uses these results to discuss the limitations of current models in long-horizon planning. To what extent can two-step planning tasks support such conclusions? Could the authors include experiments with longer action sequences to evaluate model performance across different planning horizons?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer fd8C

We thank Reviewer fd8C for recognizing the value of extending materials-science evaluation to dynamic physical evolution and intervention-based decision-making. We respond to the questions below.

W1:

We thank the reviewer for raising this important point. We clarify the scope and design of PolyWorld below.

Offline benchmark scope. We agree that PolyWorld is offline and short-horizon, but offline evaluation with fixed tasks and rule-based or ground-truth verification is a common practice in influential agent benchmarks[1,2], which have been widely used to evaluate multi-step reasoning, tool use, and action selection. PolyWorld similarly tests a verifiable capability, reasoning about intervention consequences from observed physical evolution, with labels reproducibly derived from simulation outcomes and predefined rules. We will clarify this positioning and include interactive, longer-horizon evaluation in our future work.

Chart-reading concern. To test whether performance depends on physical understanding rather than general chart interpretation, we add a current-state-only control on a fixed stratified subset of 200 Decision and Planning questions (50 each for D1/D2/P1/P2) using GPT-5.6 Sol. In this control, the current conformation, conditions, and candidate options are unchanged, but the history chart are hidden.

TaskFull historyCurrent-state-onlyΔ
D127/50 (54%)19/50 (38%)+16 pp
D217/50 (34%)16/50 (32%)+2 pp
P126/50 (52%)19/50 (38%)+14 pp
P212/50 (24%)13/50 (26%)-2 pp

Removing history reduces accuracy substantially on D1 and P1 (16 and 14 pp), indicating that models use temporal evolution information rather than static chart cues alone. We will report this analysis in the revision and temper any claim not supported by the control.

W2

We thank the reviewer for pointing this out. Due to the high cost of expert annotation, human evaluation was conducted on a subset rather than the full benchmark, which is a common practice in scientific benchmarks [3–5]. We agree that comparisons should be made on the same subset. In the revision, we will report the matched 80-question comparison as the primary human–model result and report full-benchmark model accuracy separately. We will revise the corresponding conclusions and add further analysis accordingly.

Q1

We analyze GPT-5.6 Sol's pressure-choice predictions on D2 and P2 and report the distributions below. As shown, the model exhibits a strong systematic preference for 1 atm, especially on P2, which likely contributes to its below-random accuracy. The corresponding random-guessing baselines are 33.1% for D2 and 33.3% for P2, since not all items have four options. We will add further analysis in the revision.

D2 pressure-choice distribution (226 incorrect answers)

ModelAll answers (10 atm / 1 atm / NVT)Incorrect answers (10 atm / 1 atm / NVT)
GPT-5.6 Sol4 (1.4%) / 193 (66.3%) / 94 (32.3%)1/226 (0.4%) / 154/226 (68.1%) / 71/226 (31.4%)

P2 pressure-choice distribution (202 incorrect answers)

ModelAll answers (10 atm / 1 atm / NVT)Incorrect answers (10 atm / 1 atm / NVT)
GPT-5.6 Sol28 (10.9%) / 227 (88.3%) / 2 (0.8%)14/202 (6.9%) / 186/202 (92.1%) / 2/202 (1.0%)

Q2

We will revise the corresponding claims to explicitly scope our conclusions to multi-step planning under controlled interventions, rather than implying long-horizon planning capability. We will also discuss longer action sequences and long-horizon planning as future work.

We hope the above response can further address your concern. If you have any further questions, we are pleased to discuss them.

[1] Mialon, Grégoire, et al. "Gaia: a benchmark for general ai assistants." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." International Conference on Learning Representations. Vol. 2024. 2024.

[3] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., ... & Chen, W. (2024). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9556-9567).

[4] Mirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., ... & Jablonka, K. M. (2025). A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, 17(7), 1027-1034.

[5] Lai, Z., Zheng, Y., Cai, Z., Lyu, H., Yang, J., Liang, H., ... & Wang, B. (2025). Can multimodal llms see materials clearly? a multimodal benchmark on materials characterization. arXiv preprint arXiv:2509.09307.

AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear reviewer,

We are wondering if our response and revision have resolved your concerns. If our response has addressed your concerns, we would highly appreciate it if you could re-evaluate our work and consider raising the score.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Thank you for the detailed response. It has addressed my questions and concerns. I will consider raising my score.

AuthorsOpenReview ↗

Reply to Reviewer fd8C

Thank you for reconsidering our work and raising your score! We truly appreciate your helpful feedback, which has helped us improve our paper.

Reviewer_kjUUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PolyWorld, a benchmark of 4,178 multimodal questions about polymer structures, physical evolution, and intervention outcomes. It uses controlled molecular dynamics trajectories to evaluate perception, reasoning, decision-making, and planning, with answers derived from construction records and simulated outcomes. Evaluation includes 13 MLLMs, thinking-mode and text-only ablations, and an 80-question human comparison.

Critical strengths
  1. The paper extends polymer evaluation to reasoning about intervention outcomes and subsequent action choices. This is a relevant setting for scientific agents that need to anticipate how a material will evolve before deciding what to do next.

  2. The benchmark combines diverse polymer architectures with controlled trajectory branches from shared initial states. Answers derived from construction records and simulated outcomes provide traceable references and support comparisons between alternative interventions.

Critical weaknesses
  1. The paper provides limited insight into why its strongest models fail. The analysis describes recurring confusions and presents qualitative cases, but does not systematically quantify failure patterns or distinguish perceptual errors from failures in temporal reasoning and action selection. It remains unclear which underlying limitations explain the low scores and which limitations future methods should address.

  2. The validity of the decision and planning tasks is insufficiently established. Gold actions are selected from hidden simulated futures, but the paper does not adequately demonstrate that the supplied observations support a unique and stable choice. The outcome criteria and their robustness to simulation variability are also insufficiently documented, making it difficult to distinguish failures of physical reasoning from uncertainty in the tasks themselves.

Questions for authors

Please see the Critical Weaknesses section for my main concerns.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer kjUU

We thank Reviewer kjUU for the positive comments on our intervention-based construction and the traceability of our answers.

W1

We agree that the original analysis was largely qualitative. We conducted a systematic failure analysis on 100 incorrect predictions each from Qwen3.6-27B and GPT-5.6 Sol. All 200 items were annotated with Gemini 3.6 Flash and reviewed by human experts.

Error categoryQwen3.6-27BGPT-5.6 Sol
Perception error35%22%
Temporal error10%10%
Mechanistic error37%47%
Action selection error13%21%
Format/empty response5%0%

Mechanistic and perceptual errors are the largest categories, while temporal errors are less frequent (10% for both). Action-selection errors are separately identified (13% and 21%). These results indicate that low scores are driven mainly by mechanistic and perceptual limitations, which are the main bottlenecks for future methods. We will include the full annotation protocol in the revision.

W2

We address the four concerns below.

Unique gold choice. Gold actions are not manually selected. For each item, the gold label is computed by a fixed rule over simulated candidate trajectories. In D1, we set an item-specific localization constraint at the midpoint of a gap of at least 0.025 between candidate localization values. The gold is the feasible candidate with the highest final extension, and we require a margin of at least 0.05 over the runner-up. This ensures a clear feasible/infeasible distinction and a unique best action. Full rules and worked examples will be documented in the revision.

Stability across simulation variability. Our goal is to evaluate reasoning about dynamic scientific processes, so what matters is consistent ranking of candidate actions under the same simulation setup, not invariance of absolute values across seeds. The MD pipeline is deterministic without stochastic thermostats or random forces; a seed mainly affects the initial state and initial-velocity generation. Changing the seed defines a different item, so gold stability is not defined as cross-seed reproduction. Within each item, all candidate actions share the same seed and protocol, so the seed is a common factor and relative ranking is driven by the intervention. We further retain an item only when the gold–runner-up margin is sufficiently large, so small numerical perturbations do not change the ordering. Thus, the gold label is stable within the item’s simulation setup.

Outcome criteria, robustness, and failure attribution. Items are retained only when the gold–runner-up margin is sufficiently large. This margin filtering also provides robustness, because small numerical noise does not change the ranking. Since retained items are well-separated, the low scores are not simply due to ambiguous tasks. Our failure analysis (above) shows that errors concentrate in mechanistic and perceptual categories, and human experts perform substantially better on the matched subset. These results indicate that model failures reflect physical-reasoning limitations rather than task uncertainty.

Below-random accuracy on D2 and P2. We further analyzed whether the below-random accuracy on these subtasks (random baselines of 33.1% and 33.3%, respectively) is associated with option-level preferences. We find that GPT-5.6 Sol systematically favors the 1 atm option. In D2, 66.3% of all answers and 68.1% of incorrect answers select 1 atm. In P2, the proportions are 88.3% and 92.1%. This option-level bias likely contributes to the below-random accuracy and helps explain the low scores on these subtasks.

We hope this addresses the reviewer’s concerns.

AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear reviewer,

We are wondering if our response and revision have resolved your concerns. If our response has addressed your concerns, we would highly appreciate it if you could re-evaluate our work and consider raising the score.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Thank you for the response. It addresses most of my concerns and I will raise my score.

AuthorsOpenReview ↗

Reply to Reviewer kjUU

Thank you for reconsidering our work and for raising your score! We sincerely appreciate your valuable feedback, which has helped us further improve our paper.

Reviewer_tfGROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PolyWorld evaluates multimodal reasoning over polymer structures and simulated physical evolution. It contains 4,178 questions spanning perception, reasoning, decision-making, and planning. Molecular dynamics simulations generate alternative intervention outcomes, from which predefined rules determine reference answers. The paper evaluates 13 models. The best model achieves 44.28% overall accuracy, with weaker performance on trade-off decisions and adaptive planning.

Critical strengths
  1. The intervention-based construction is useful. Simulating alternative actions from a shared state supports questions about action consequences. This extends evaluation beyond static material recognition and property prediction.

  2. The answers are traceable to underlying data. Construction records and simulated observables provide a basis for programmatic labeling. This can support reproducible evaluation and reduce dependence on subjective answer annotation.

Critical weaknesses
  1. The decision and planning tasks are insufficiently specified. Appendix A.1 describes the intervention branches but does not provide the exact objectives, instability thresholds, or tie-handling rules used to select answers. It also remains unclear whether the visible observations contain enough information to identify the best hidden future outcome. These details are central to judging whether the questions are well-posed.

  2. The evaluation does not fully support the main capability claims. The headline human–model comparison mixes human accuracy on 80 questions with model accuracy on the full benchmark. On the matched subset, the scores are 71.25% and 58.75%. The text-only ablation is also affected by increased empty responses. Without temporal controls, the results do not establish how much performance depends on reasoning over evolution rather than static visual evidence.

Questions for authors
  1. Can you provide a complete decision and planning example, including all model-visible inputs, the scoring formula, constraint thresholds, and the computed outcomes for every action?

  2. How stable are the selected answers across simulation seeds? Are questions with small outcome differences or near-tied actions excluded?

  3. Can you report uncertainty for the matched human–model comparison and add temporal controls, such as shuffled frames or current-state-only input? Please also separate empty-response failures from substantive errors in the text-only ablation.

5 comments
AuthorsOpenReview ↗

Reply to Reviewer tfGR

We thank Reviewer tfGR for recognizing the value of our intervention-based construction and the traceability of our answers. Below, we respond to the questions.

W1:

Thanks for noting that our description of the Decision and Planning tasks needs more detail. We will clarify the objectives and answer-selection rules in the revision.

Exact objectives used to select answers. We focus on extension and localization, both essential in polymer research.

  • D1: maximize final extension among candidates whose localization is below an item-specific limit. The limit is set at the midpoint of a gap of at least 0.025 between two candidate localization values, and candidates with localization below this limit satisfy the constraint. The options include the top two candidates satisfying the constraint and the unconstrained highest-extension candidate.

  • D2: maximize

U(a)=Efinal(a)−λmax⁡(0,Lmax(a)−L0).U(a)=E_{\rm final}(a)-\lambda\max(0,L_{\rm max}(a)-L_0).

Here aa is a candidate intervention, U(a)U(a) is its utility, Efinal(a)E_{\rm final}(a) is the final relative extension under aa, Lmax(a)L_{\rm max}(a) is the maximum localization along the candidate trajectory, L0L_0 is the localization at the decision point, and λ\lambda is the penalty weight for localization growth. The term max⁡(0,Lmax(a)−L0)\max(0,L_{\rm max}(a)-L_0) penalizes only localization growth beyond L0L_0. Thus, D2 selects the action with the largest final extension after penalizing excessive localization growth.

  • P1: select the complete action sequence with highest final extension.

  • P2: after observing the intermediate state, select the next intervention that maximizes final extension.

Instability thresholds. A candidate is marked runaway if Lpeak>1.2L_{\rm peak}>1.2 or ∣Efinal∣>2|E_{\rm final}|>2. Here LpeakL_{\rm peak} is the maximum localization along the trajectory and EfinalE_{\rm final} is the final relative extension. This exclusion prevents extreme or numerically anomalous simulations from becoming gold solely due to large extension gain.

Tie-handling rules. We avoid ambiguous ties by filtering near-ties with explicit margins and excluding exact numerical ties. We primarily use a margin of 0.05 for both decision and planning tasks. Near-identical outcomes are flagged for review, so gold answers do not rely on arbitrary tie-breaking.

Visible observations and well-posedness. Each item provides the current conformation, the observed history of extension (EE), localization (LL), and alignment (SS) up to the decision point, the experimental conditions, and the candidate interventions. Candidate futures and outcome scores are hidden. The gold answer is defined by applying the above objective and selection rule to the simulated outcomes for that fixed instance.

W2 & Q3:

We thank the reviewer for these constructive comments. We address the three concerns in order below.

Matched human–model comparison. Due to the high cost of expert annotation, we asked experts to answer only a subset of the benchmark, which is a common practice in scientific benchmarks [1,2,3]. To ensure a fair comparison, we report McNemar exact tests for the main proprietary models below. Most baselines, including strong proprietary models, have p < 0.05, and the only exception is DeepSeek-V4-Flash-Vision. The complete table will be included in the revision, and we will revise the corresponding claims in the revision to make them more precise.

GPT-5.6 TerraGPT-5.6 SolGemini 3.6 FlashClaude Sonnet 4.6DeepSeek-V4-Flash-Vision
Exact p0.0008786.62×10⁻⁵0.001070.009040.1214

Text-only ablation with empty responses separated. To exclude the impact of empty responses, we report a paired non-empty analysis below. In this analysis, full-input accuracy is recomputed on the same subset of questions that received non-empty text-only responses. As shown, even after excluding empty responses, full-input accuracy remains higher than text-only accuracy for all three models, including Qwen3.6-27B, which produces no empty responses. This indicates that the visual advantage is not solely driven by empty responses. We will further clarify this analysis in the revision.

ModelEmpty rateFull (paired non-empty)Text-only (paired non-empty)Δ
Qwen3.6-27B0.00%40.45%32.67%+7.78
Claude Sonnet 4.643.25%46.94%35.34%+11.60
GPT-5.6 Terra47.53%38.87%30.52%+8.35
AuthorsOpenReview ↗

Reply to Reviewer tfGR

Temporal controls.

We add a current-state-only control on a fixed stratified subset of 200 Decision and Planning questions (50 each for D1/D2/P1/P2) on GPT-5.6 Sol. We compare full-history accuracy against current-state-only accuracy below.

TaskFull historyCurrent-state-onlyΔ
D127/50 (54%)19/50 (38%)+16 pp
D217/50 (34%)16/50 (32%)+2 pp
P126/50 (52%)19/50 (38%)+14 pp
P212/50 (24%)13/50 (26%)-2 pp

Removing history causes an overall 7.5 pp drop (42% vs. 33.5%). The drop is especially large on D1 and P1 (16 and 14 pp), while D2 and P2 remain similar. This indicates that temporal history is useful, particularly for D1 and P1. We will add this analysis to the revision.

Q1:

Below we provide complete D1 and P2 examples with model-visible inputs, scoring rules, thresholds, and per-action outcomes.

D1 (Constrained Decision) example

Model-visible inputs. The model sees the current polymer conformation, the observed E/L/SE/L/S history up to the decision point, the experimental conditions, and four candidate two-step action plans:

  • A: continue stretching, then continue stretching
  • B: hold box dimensions, then continue stretching
  • C: continue stretching, then hold box dimensions
  • D: continue stretching, then unload

Scoring rule and threshold. For each candidate plan, we compute the maximum of the three-frame moving average of localization, LpeakL_{\rm peak}. A plan is feasible if Lpeak≤τL_{\rm peak}\le\tau; among feasible plans, we select the one with the highest final extension EfinalE_{\rm final}. For this item, the evaluator threshold is τ=0.6272\tau=0.6272.

Computed outcomes.

PlanFinal ELpeakL_{\rm peak}Feasible
A: continue, continue-0.38180.6057yes
B: hold, continue-0.17310.6856no
C: continue, hold-0.50540.6057yes
D: continue, unload-0.62910.6057yes

B has the highest final E but violates the localization constraint. Among feasible plans, A has the highest final E, so the gold answer is A. The final-E margin between A and the runner-up C is 0.1236.

P1 (Look-ahead Planning) example

Model-visible inputs. The model sees the current polymer conformation and the observed E/L/SE/L/S history up to the decision point. The question states that intermediate states and final outcomes after the second step are hidden. The model must choose a complete two-step plan that maximizes final extension EfinalE_{\rm final}. The four options are:

  • A: continue stretching, then hold box dimensions
  • B: continue stretching, then continue stretching
  • C: unload, then continue stretching
  • D: hold box dimensions, then continue stretching

Scoring rule. This item has no localization cap and no penalty term. The gold answer is simply the plan with the highest final extension, i.e., arg⁡max⁡Efinal\arg\max E_{\rm final}.

Computed outcomes.

OptionPlanFinal E
ACONT → HOLD1.0932
BCONT → CONT1.6165
CUNLOAD → CONT1.1967
DHOLD → CONT0.9222

The gold answer is B. Its final E exceeds the runner-up C by 0.4198.

Item-filtering criteria (not a model-facing constraint). These are construction-time filters used to remove items with insufficient separation, not physical constraints that the model must apply. An item is retained only if (i) different first-step actions produce sufficiently different intermediate states, with ∣ΔL∣≥0.04|\Delta L|\ge 0.04 or ∣ΔE∣≥0.08|\Delta E|\ge 0.08; (ii) the best complete plan exceeds the runner-up in final E by at least 0.05; and (iii) the item is excluded if the first action alone already determines the full answer. In this example, the first-step branches differ, the best plan exceeds the runner-up by 0.4198, and the item is marked as passing the first-step-shortcut check.

Q2:

We clarify the role of simulation seeds in PolyWorld. The MD pipeline uses fixed force-field parameters and deterministic integration without stochastic thermostats or random forces. A seed affects the initial state and initial-velocity generation, not the intervention protocol applied afterward. Importantly, for each Decision or Planning item, all candidate actions branch from the same decision-point state and share the same seed and simulation protocol. Thus, the seed is a common factor across candidates, while the relative ranking of candidates is driven by the intervention itself.

For near-tied actions, as described in our response to W1, we retain an item only when the gap between the gold and runner-up outcomes is sufficiently large. Items with small outcome differences are filtered or flagged for review. Therefore, the gold labels are defined on items with clear outcome separation.

We hope the above response can further address your concern. If you have any further questions, we are pleased to discuss them.

AuthorsOpenReview ↗

Reply to Reviewer tfGR

References

[1] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., ... & Chen, W. (2024). Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9556-9567).

[2] Mirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., ... & Jablonka, K. M. (2025). A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, 17(7), 1027-1034.

[3] Lai, Z., Zheng, Y., Cai, Z., Lyu, H., Yang, J., Liang, H., ... & Wang, B. (2025). Can multimodal llms see materials clearly? a multimodal benchmark on materials characterization. arXiv preprint arXiv:2509.09307.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Comment to all reviewers

We thank the reviewers for finding our intervention-based construction “useful” and our answers “traceable” (Reviewer tfGR), our setting “relevant for scientific agents” (Reviewer kjUU), and our benchmark “systematic and grounded in physical simulations” with “significant research value” (Reviewer fd8C). We would like to express our gratitude for the valuable feedback, which has significantly strengthened our paper (with the revision now posted).

In response to concerns raised by multiple reviewers regarding the specification and validity of Decision and Planning tasks, we added construction details in Appendix A.4. For the human–model comparison, we revised the headline analysis in Section 4.2 and Appendix A.5, and report the matched 80-question subset result with McNemar exact tests.

We noted that reviewers also raised concerns regarding the lack of systematic analysis. We added a current-state-only control in Appendix A.6, showing that removing history reduces accuracy on Constrained Decision and Look-ahead Planning by 16 and 14 pp, confirming the use of temporal evolution information. We further added a failure taxonomy on 200 incorrect predictions and an option-level preference analysis explaining the below-random accuracy on Trade-off Decision and Adaptive Planning.

We also revised the Limitations section and added a Future Work section. We hope that with these changes, we address all major concerns.

AuthorsOpenReview ↗

Response Summary

We sincerely thank the reviewers, Area Chairs (ACs), Senior Area Chairs (SACs), and Program Chairs (PCs) for their time and effort throughout the review process. We are particularly grateful for the reviewers' insightful comments and helpful suggestions.

As the rebuttal period comes to a close, we would like to briefly summarize the reviewers' feedback and the revisions made to the paper.


We appreciate the reviewers' recognition of the contributions of our work:

  • Reviewer tfGR:
    • "the intervention-based construction is useful", "answers are traceable to underlying data", "supports reproducible evaluation".
  • Reviewer kjUU:
    • "extends polymer evaluation to reasoning about intervention outcomes and subsequent action choices", "a relevant setting for scientific agents", "diverse polymer architectures with controlled trajectory branches".
  • Reviewer fd8C:
    • "extends the scope of evaluation to include dynamic physical evolution and intervention-based decision-making", "significant research value", "systematic and grounded in physical simulations", "objectivity and reproducibility of the evaluation".

In response to the reviewers' comments, we have made the following revisions:

1. Task Construction and Validity

  • As suggested by Reviewers tfGR and kjUU, we clarified task objectives, evaluation criteria, and simulation protocols, including tie handling and the use of shared initial states.

2. Experiments and Analysis

  • As suggested by Reviewers tfGR and fd8C, we revised the human–model comparison and added statistical significance tests.
  • As suggested by Reviewers tfGR and fd8C, we added text-only and current-state-only ablations to examine the contributions of visual inputs and historical information.
  • As suggested by Reviewers kjUU and fd8C, we added failure analysis and option-level preference analysis to better understand model limitations.

3. Scope and Clarity

  • As suggested by Reviewer fd8C, we expanded the discussion of limitations and future directions, highlighting the need for longer-horizon and interactive evaluation in future work.

We appreciate the positive responses to our revisions and hope that the additional experiments and clarifications summarized above also address the concerns raised by Reviewer tfGR, many of which overlap with those raised by the other reviewers.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.