acceptodds
Under review as a conference paper at ICLR 2027

ChemArena: Evaluating Task-Specific Reasoning Requirements in Multimodal Chemical Reasoning

Abstract

We present ChemArena, a task-aware benchmark for evaluating observable reasoning requirements in multimodal chemical reasoning. It contains 1,033 instances spanning five representative chemical task families, organizing three observable reasoning requirements: Evidence Utilization, Constraint Adherence, and Cross-Turn State Maintenance. To evaluate these task-specific requirements, ChemArena introduces a requirement-aware evaluation protocol that integrates task-specific outcome verification, reference-guided process alignment, and context-enriched reasoning assessment. Experiments on 16 representative MLLMs demonstrate that models with similar task-level performance can exhibit substantially different reasoning behaviors across chemical scenarios. ChemArena provides a systematic framework for analyzing multimodal chemical reasoning beyond accuracy-based evaluation and enables fine-grained understanding of model capabilities and limitations.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Reasoning Evaluation, Benchmark, Requirement-Aware Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aDVyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChemArena, a benchmark for evaluating multimodal chemical reasoning beyond final-answer accuracy. It contains 1,033 evaluation units across five chemical task families, organized around three reasoning requirements: evidence utilization, constraint adherence, and cross-turn state maintenance. The authors propose a three-stage evaluation protocol combining task-specific outcome verification, reference-guided process alignment, and an LLM judge calibrated against human annotations. Experiments on 16 MLLMs show that models with similar overall scores can have quite different strengths and weaknesses across these reasoning requirements.

Critical strengths
  1. The paper addresses an important limitation of existing chemistry benchmarks. A correct final answer does not necessarily mean that a model has used the right chemical evidence or followed the required constraints. Evaluating these aspects separately is a useful direction.
  2. The three reasoning requirements are intuitive and reasonably well motivated. I particularly like the distinction between evidence utilization, constraint adherence, and cross-turn state maintenance, since these capture different challenges in chemical reasoning.
  3. The evaluation protocol combines deterministic verification with LLM-based assessment. Using chemical tools for verifiable properties, rather than relying entirely on an LLM judge, is a sensible design choice.
  4. The evaluation covers 16 models with different capabilities and reasoning configurations. The results provide some interesting observations, especially that models with similar overall performance can behave quite differently across the three requirements.
  5. The authors provide substantial implementation and annotation details in the appendix. The human calibration of the LLM judge and the qualitative failure cases are also useful attempts to make the evaluation more interpretable.
Critical weaknesses
  1. Validation of the reasoning evaluation. Although the authors calibrate the LLM judge against human scores, I would like to see stronger evidence that the framework correctly identifies reasoning failures. In particular, the accuracy of the failure diagnosis itself is not directly evaluated against expert annotations. One example that confused me is Figure 5, where the diagnosis mentions an "unsupported kinetic inference," while the displayed reasoning concerns equilibrium selectivity rather than reaction rates. This makes me wonder how reliable the generated diagnoses are.
  2. Dependence on reference reasoning traces. Stage II relies on matching model responses to human-verified checkpoints. While the paper allows alternative reasoning paths, I am not fully convinced that the matching procedure handles them reliably. A model could provide a correct but shorter explanation without explicitly covering all reference checkpoints. It would be helpful to test how sensitive the scores are to different valid reasoning traces or levels of explanation detail.
  3. Limited evaluation of the proposed protocol itself. The paper presents a fairly elaborate three-stage evaluation framework, but I could not find a direct comparison with simpler evaluation approaches. For example, how much does Stage II actually improve the final reasoning assessment compared with using an LLM judge directly? An ablation of the evaluation stages would help establish the value of the proposed design.
  4. Annotation and judge reliability. The authors describe a careful annotation process, but some useful information is still missing. In particular, the judge calibration subset size is not clearly reported, and inter-annotator agreement for the reference annotations is unavailable. I appreciate that the authors acknowledge this limitation, but these details matter for a benchmark that relies heavily on human-verified reasoning checkpoints.
  5. Dataset coverage and score interpretation. The benchmark covers several tasks, but the three requirements are represented by quite different types and amounts of data. For example, the cross-turn evaluation contains only 42 dialogues, and not all turns have process annotations. Also, the Overall score gives equal weight to the three requirements. It would be useful to discuss how these choices affect the model rankings and how well the findings generalize to other chemical reasoning settings.
Questions for authors
  1. Could the authors clarify how the accuracy of the failure diagnosis is validated? In particular, would it be possible to compare the identified reasoning failures with independent expert judgments on a subset of examples?
  2. Could the authors provide an ablation comparing the full three-stage evaluation with a simpler LLM-as-Judge approach? I would also be interested in whether the main model rankings remain consistent when using different valid reference traces or judge models.
1 comment

We have replaced the original Figure 5 example with a molecular-editing case whose generated structure visibly violates the requested transformation constraints. Figure 4 illustrates an Evidence Utilization failure: the answer is correct, but the reasoning does not connect the visual evidence to the conclusion. Section 3.4 and Appendix D.4 explain how task requirements, reference checkpoints, and verification and alignment records guide failure diagnosis.

Section 3.4 and Appendix C.4 clarify that reference checkpoints are evaluation anchors. Chemically equivalent terminology, concise derivations, and alternative valid decompositions or step orders are accepted when they satisfy the required evidence, constraints, and prerequisite relations. Stage II records checkpoint realization and dependencies using lexical variants, structured aliases, and task-local rules. Stage III assesses chemical validity and sufficiency in the original task context. Process Alignment and Reasoning Quality are reported separately to distinguish checkpoint coverage from explanation quality.

Appendix D.1 and Table 11 compare four protocols on the full validation set of 944 responses from 118 evaluation samples. Each protocol evaluates the same saved responses three times. Against the mean of eight human ratings, the full protocol achieves MAE 0.777 ± 0.020, compared with 0.879 ± 0.020 for direct single-pass judgment, 0.883 ± 0.021 for reference-informed single-pass judgment, and 0.852 ± 0.020 without Alignment Context. Values are point estimates ± standard deviations; Appendix D.1 describes their calculation. These results demonstrate improved agreement with human judgments and the contribution of Stage II alignment information.

To assess ranking sensitivity to judge choice, we randomly sampled 103 of the benchmark's 1,033 evaluation instances (approximately 10%) and evaluated nine models with Gemini 3 Flash. All nine models used the same subset: 82 single-turn questions and 21 turns from four complete dialogues. Appendix E.9 and Tables 15–18 report the results. The Gemini 3 Flash subset Overall ranking and GPT-5.4-mini full-benchmark ranking show Spearman correlation 0.883 and Kendall correlation 0.778, supporting broadly consistent overall rankings across the two evaluation settings.

Appendix E.1 and Tables 12–14 describe the calibration set and human-rating reliability. The set comprises 118 evaluation samples: 108 single-turn questions and 10 complete dialogues. Eight models contribute 944 responses, each scored by eight chemistry-background raters, yielding 7,552 ratings. Overall ordinal Krippendorff's alpha is 0.530, ICC(A,1) is 0.559, and ICC(A,8) is 0.910. Table 14 also reports task-specific estimates and 95% cluster-bootstrap intervals. Section 3.3 and Appendix C.6 describe independent reference annotation by two chemistry-trained annotators, cross-checking, and expert review and adjudication.

Section 3.3, Appendix C.5, and Table 9 specify 42 complete dialogues containing 215 evaluated turns, including 125 turns with process annotations (58.1%). These annotations require expert verification of chemical reasoning, cross-turn state dependencies, and structured checkpoints, taking approximately four hours per expert per dialogue. All 215 turns contribute to outcome evaluation; Stage II uses the 125 turns with complete expert-verified process references. Benchmark-wide process annotation coverage is 943 of 1,033 evaluation instances (91.3%).

Appendix D.5 defines each requirement-axis composite as 0.33 Outcome + 0.33 Alignment + 0.34 Reasoning Quality, and Overall as the macro-average of the three axis composites. Each reasoning requirement therefore has equal influence on Overall. Requirement-level and component-level scores accompany Overall to make model comparisons explicit across CHEMARENA's five task families and three reasoning requirements.

Reviewer_xaboOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ChemArena organizes five multimodal chemistry task families around three observable reasoning requirements: Evidence Utilization, Constraint Adherence, and Cross-Turn State Maintenance. Its three-stage evaluation protocol separates deterministic task-outcome verification, reference-guided process alignment, and context-enriched reasoning assessment by a human-calibrated LLM judge. The authors evaluate 16 multimodal models on 1,033 evaluation units, comprising 818 single-turn instances and 215 turns from 42 dialogues. The results show that models with similar task-level performance can exhibit different observable reasoning behaviors.

Critical strengths
  1. The benchmark construction pipeline incorporates source validation, independent annotation, cross-checking, and expert adjudication. It covers complementary task families and reports component-level results and annotation procedures. Appendix E.1 explicitly excludes the judge-calibration development subset from final benchmark reporting, providing an important evaluation control.
  2. Separating chemical outcomes, process alignment, and semantic assessment provides a useful framework for examining failures that final-answer accuracy can miss. Several task outcomes are verified using deterministic chemical checks independently of the LLM judge.
Critical weaknesses
  1. Judge-calibration reporting is incomplete. Appendix E.1 specifies eight independent human ratings per response but does not report the calibration sample size, task and response-model composition, inter-rater reliability, or uncertainty estimates. (btw the Table 2 and Table 11 are duplicated) The existing table values appear to correspond to a very small calibration set. The authors should clarify the actual size and composition of this subset. Appendix E.9 also states that Gemini 3 Flash is retained as an auxiliary judge for ranking sensitivity analysis, but the corresponding results are not reported in the paper.
  2. The annotation quality-control procedure lacks quantitative reliability evidence. Appendix C.6 describes independent annotation, cross-checking, and expert adjudication, but explicitly states that pooled inter-annotator agreement and adjudication rates are unavailable because the early records were not consolidated.
  3. The contribution of the three-stage protocol has not been established through direct comparisons and controlled experiments. Section 3.4 and Appendix D describe how reference alignment, task reconstruction, reasoning audit, and failure diagnosis pass contextual information to subsequent stages. However, the experiments primarily consist of model score tables, capability profiles, qualitative case studies, and candidate-judge calibration. These results do not isolate whether the full protocol improves agreement with human judgments or error-localization accuracy relative to a single-pass judge given the same task and reference information, or whether Stage II’s deterministic alignment improves subsequent judgments. A direct comparison and ablations of the Outcome Context, Alignment Context, and task reconstruction step would help establish the contribution of these components.
Questions for authors
  1. What are the size, sampling procedure, task composition, and response-model composition of the judge-calibration subset? How consistent are the eight human raters, and what independent human validation supports the diagnostic scores? Please also report the ranking sensitivity results obtained with Gemini 3 Flash.
  2. Can an independent sample audit quantify the reliability of the final reference annotations? On the same human-annotated evaluation set, how does the full protocol compare with a single-pass judge and variants that omit the key contextual inputs?
1 comment

Appendix E.1 and Tables 12–14 report the calibration-data composition, sampling procedure, and human-rating reliability. The calibration data contain 118 evaluation samples: 108 single-turn questions and 10 complete dialogues containing 52 turns. Each evaluation sample corresponds to one question or one complete dialogue. Evaluation samples were randomly sampled within each task family after specifying the required counts. Eight models, listed in Table 13, each contribute one model response per evaluation sample, yielding 944 model responses. Each model response is scored by eight chemistry-background raters, giving 7,552 human ratings. For multi-turn tasks, each model response contains the output for a complete dialogue and receives dialogue-level ratings. Score-agreement metrics are computed over the 944 model responses (N = 944), pairing each judge score with the mean of eight human ratings for that response. Overall ordinal Krippendorff's alpha is 0.530, ICC(A,1) is 0.559, and ICC(A,8) is 0.910; Table 14 also reports task-specific estimates and 95% cluster-bootstrap intervals, using questions or complete dialogues as the resampling clusters. We have corrected the table duplication: Table 2 reports candidate-judge calibration, whereas Table 11 reports protocol ablation. To examine potential same-family scoring preferences when GPT-5.4-mini evaluates GPT-family models, we additionally sampled 103 evaluation units at random from the full benchmark of 1,033 evaluation units (approximately 10%) and evaluated nine models using Gemini 3 Flash as an auxiliary judge. The same subset was used for all nine models and comprises 82 single-turn questions and 21 turns from four complete dialogues. Appendix E.9 and Tables 15–18 report the results. Across these nine models, the Gemini subset Overall ranking and the GPT-5.4-mini full-benchmark ranking have Spearman correlation 0.883 and Kendall correlation 0.778, showing broadly consistent overall rankings and providing an additional cross-family check on the model comparisons. These results further support the fairness of GPT-5.4-mini as an evaluation model, including when assessing responses from GPT-family models. Section 3.3 and Appendix C.6 detail the verification procedure for the final reference annotations. Two chemistry-trained annotators independently construct the reference rationale and structured checkpoints, followed by cross-checking for chemical validity, checkpoint completeness, dependency consistency, and agreement with the reference answer. Disagreements are resolved through discussion and, when necessary, adjudication by a professor or associate professor with relevant chemistry expertise. Expert review verifies the annotation structure and chemical consistency before finalization. This procedure provides checks at both the reasoning-content and structured-checkpoint levels. Appendix D.1 and Table 11 directly compare four variants on the same saved model responses: direct single-pass judgment, reference-informed single-pass judgment, the full three-stage protocol, and the full protocol without Alignment Context. Each variant is evaluated three times on the same saved responses. Values below are full-validation-set point estimates ± cluster-bootstrap standard deviations. Against the mean of eight human ratings, the full protocol achieves the lowest MAE (0.777 ± 0.020), compared with 0.879 ± 0.020 for direct judgment and 0.883 ± 0.021 for reference-informed judgment. Removing Alignment Context increases MAE to 0.852. The full protocol also achieves the highest Pearson correlation (0.668 ± 0.018), Spearman correlation (0.666 ± 0.017), and within-question pairwise ranking accuracy (0.681 ± 0.013) among the four variants. These results support the full protocol’s improved agreement with human judgments and the contribution of Stage II’s Alignment Context to subsequent assessment.

Reviewer_djbGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChemArena, a benchmark for multimodal chemical reasoning. It contains 1,033 evaluation units from five task families and organizes them around three proposed observable reasoning requirements: Evidence Utilization, Constraint Adherence, and Cross-Turn State Maintenance. The evaluation protocol combines task-specific outcome verification, reference-guided process alignment, and LLM-judge-based reasoning assessment. Experiments on 16 MLLMs show that similar outcome scores can correspond to different process-level profiles.

Critical strengths
  1. The paper addresses an important gap in chemical MLLM evaluation. Separating answer correctness from observable reasoning failures is valuable, particularly for chemistry tasks where a chemically plausible final output may still violate a structural constraint or misuse evidence.
  2. The benchmark covers diverse settings, including visual chemistry questions, molecular editing and optimization, property prediction, and multi-turn tasks. The combination of deterministic chemical verification, structured process annotations, and qualitative failure reports is a useful and thoughtful design.
Critical weaknesses
  1. The main contribution appears to be the integration of existing chemical tasks, human reference rationales, rule-based process matching, and LLM-as-a-Judge evaluation into a three-stage pipeline. While this integration may be practically useful for chemical applications, the paper does not yet clearly establish a substantive methodological advance over existing process-oriented benchmarks.
  2. The benchmark has limited sample sizes in several task families. Although it reports 1,033 evaluation units, the multi-turn subset contains only 42 dialogues.
Questions for authors

Please release the exact prompts, image preprocessing procedure, model/API versions, inference dates, decoding settings, retry policy, and evaluation code. Were models evaluated once or over multiple stochastic runs?

1 comment

We have clarified the contribution in Sections 2 and 3. CHEMARENA operationalizes task-specific reasoning requirements through distinct alignment structures: an overall-plus-options structure for Evidence Utilization, a constraint chain for Constraint Adherence, and a turn-state graph for Cross-Turn State Maintenance. Structured verification and alignment records inform subsequent semantic assessment, while outcome performance, process alignment, and reasoning quality remain separately reported. Appendix A, Table 3, compares its evaluation coverage with existing chemistry benchmarks. To assess the value of this integrated design, Appendix D.1, Table 11, compares evaluation protocols on the same saved responses. Against the mean of eight human ratings, the full protocol achieves an MAE of 0.777 ± 0.020, compared with 0.879 ± 0.020 for direct single-pass judgment and 0.883 ± 0.021 for reference-informed single-pass judgment, indicating improved score agreement in this comparison. The multi-turn subset comprises 42 complete dialogues with 215 evaluated turns. Constructing the process annotations requires expert verification of chemical reasoning, cross-turn state dependencies, and structured checkpoints, taking approximately four hours per expert per dialogue on average. Process alignment is evaluated on the 125 turns (58.1%) with complete expert-verified reference trajectories and checkpoints, while all 215 turns contribute to outcome evaluation. Section 3.3 and Table 9 clarify the dialogue-level sample size, turn-level evaluation units, and process-annotation coverage. Appendix H identifies the single-turn, multi-turn, and judge prompt templates and describes their construction. At inference time, the selected final image is read and encoded without additional resizing or cropping in the request pipeline. The revised Table 22 reports model identifiers, providers, reasoning and decoding settings, and saved date ranges spanning May 4–7, 2026. The generation script allows up to three application-level attempts per item, with a two-second delay between failed attempts. Each model generated one set of benchmark responses. The protocol comparison in Table 11 repeats judge evaluation three times on the same saved responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.