- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ChemArena, a benchmark for evaluating multimodal chemical reasoning beyond final-answer accuracy. It contains 1,033 evaluation units across five chemical task families, organized around three reasoning requirements: evidence utilization, constraint adherence, and cross-turn state maintenance. The authors propose a three-stage evaluation protocol combining task-specific outcome verification, reference-guided process alignment, and an LLM judge calibrated against human annotations. Experiments on 16 MLLMs show that models with similar overall scores can have quite different strengths and weaknesses across these reasoning requirements.
Critical strengths
- The paper addresses an important limitation of existing chemistry benchmarks. A correct final answer does not necessarily mean that a model has used the right chemical evidence or followed the required constraints. Evaluating these aspects separately is a useful direction.
- The three reasoning requirements are intuitive and reasonably well motivated. I particularly like the distinction between evidence utilization, constraint adherence, and cross-turn state maintenance, since these capture different challenges in chemical reasoning.
- The evaluation protocol combines deterministic verification with LLM-based assessment. Using chemical tools for verifiable properties, rather than relying entirely on an LLM judge, is a sensible design choice.
- The evaluation covers 16 models with different capabilities and reasoning configurations. The results provide some interesting observations, especially that models with similar overall performance can behave quite differently across the three requirements.
- The authors provide substantial implementation and annotation details in the appendix. The human calibration of the LLM judge and the qualitative failure cases are also useful attempts to make the evaluation more interpretable.
Critical weaknesses
- Validation of the reasoning evaluation. Although the authors calibrate the LLM judge against human scores, I would like to see stronger evidence that the framework correctly identifies reasoning failures. In particular, the accuracy of the failure diagnosis itself is not directly evaluated against expert annotations. One example that confused me is Figure 5, where the diagnosis mentions an "unsupported kinetic inference," while the displayed reasoning concerns equilibrium selectivity rather than reaction rates. This makes me wonder how reliable the generated diagnoses are.
- Dependence on reference reasoning traces. Stage II relies on matching model responses to human-verified checkpoints. While the paper allows alternative reasoning paths, I am not fully convinced that the matching procedure handles them reliably. A model could provide a correct but shorter explanation without explicitly covering all reference checkpoints. It would be helpful to test how sensitive the scores are to different valid reasoning traces or levels of explanation detail.
- Limited evaluation of the proposed protocol itself. The paper presents a fairly elaborate three-stage evaluation framework, but I could not find a direct comparison with simpler evaluation approaches. For example, how much does Stage II actually improve the final reasoning assessment compared with using an LLM judge directly? An ablation of the evaluation stages would help establish the value of the proposed design.
- Annotation and judge reliability. The authors describe a careful annotation process, but some useful information is still missing. In particular, the judge calibration subset size is not clearly reported, and inter-annotator agreement for the reference annotations is unavailable. I appreciate that the authors acknowledge this limitation, but these details matter for a benchmark that relies heavily on human-verified reasoning checkpoints.
- Dataset coverage and score interpretation. The benchmark covers several tasks, but the three requirements are represented by quite different types and amounts of data. For example, the cross-turn evaluation contains only 42 dialogues, and not all turns have process annotations. Also, the Overall score gives equal weight to the three requirements. It would be useful to discuss how these choices affect the model rankings and how well the findings generalize to other chemical reasoning settings.
Questions for authors
- Could the authors clarify how the accuracy of the failure diagnosis is validated? In particular, would it be possible to compare the identified reasoning failures with independent expert judgments on a subset of examples?
- Could the authors provide an ablation comparing the full three-stage evaluation with a simpler LLM-as-Judge approach? I would also be interested in whether the main model rankings remain consistent when using different valid reference traces or judge models.
We have replaced the original Figure 5 example with a molecular-editing case whose generated structure visibly violates the requested transformation constraints. Figure 4 illustrates an Evidence Utilization failure: the answer is correct, but the reasoning does not connect the visual evidence to the conclusion. Section 3.4 and Appendix D.4 explain how task requirements, reference checkpoints, and verification and alignment records guide failure diagnosis.
Section 3.4 and Appendix C.4 clarify that reference checkpoints are evaluation anchors. Chemically equivalent terminology, concise derivations, and alternative valid decompositions or step orders are accepted when they satisfy the required evidence, constraints, and prerequisite relations. Stage II records checkpoint realization and dependencies using lexical variants, structured aliases, and task-local rules. Stage III assesses chemical validity and sufficiency in the original task context. Process Alignment and Reasoning Quality are reported separately to distinguish checkpoint coverage from explanation quality.
Appendix D.1 and Table 11 compare four protocols on the full validation set of 944 responses from 118 evaluation samples. Each protocol evaluates the same saved responses three times. Against the mean of eight human ratings, the full protocol achieves MAE 0.777 ± 0.020, compared with 0.879 ± 0.020 for direct single-pass judgment, 0.883 ± 0.021 for reference-informed single-pass judgment, and 0.852 ± 0.020 without Alignment Context. Values are point estimates ± standard deviations; Appendix D.1 describes their calculation. These results demonstrate improved agreement with human judgments and the contribution of Stage II alignment information.
To assess ranking sensitivity to judge choice, we randomly sampled 103 of the benchmark's 1,033 evaluation instances (approximately 10%) and evaluated nine models with Gemini 3 Flash. All nine models used the same subset: 82 single-turn questions and 21 turns from four complete dialogues. Appendix E.9 and Tables 15–18 report the results. The Gemini 3 Flash subset Overall ranking and GPT-5.4-mini full-benchmark ranking show Spearman correlation 0.883 and Kendall correlation 0.778, supporting broadly consistent overall rankings across the two evaluation settings.
Appendix E.1 and Tables 12–14 describe the calibration set and human-rating reliability. The set comprises 118 evaluation samples: 108 single-turn questions and 10 complete dialogues. Eight models contribute 944 responses, each scored by eight chemistry-background raters, yielding 7,552 ratings. Overall ordinal Krippendorff's alpha is 0.530, ICC(A,1) is 0.559, and ICC(A,8) is 0.910. Table 14 also reports task-specific estimates and 95% cluster-bootstrap intervals. Section 3.3 and Appendix C.6 describe independent reference annotation by two chemistry-trained annotators, cross-checking, and expert review and adjudication.
Section 3.3, Appendix C.5, and Table 9 specify 42 complete dialogues containing 215 evaluated turns, including 125 turns with process annotations (58.1%). These annotations require expert verification of chemical reasoning, cross-turn state dependencies, and structured checkpoints, taking approximately four hours per expert per dialogue. All 215 turns contribute to outcome evaluation; Stage II uses the 125 turns with complete expert-verified process references. Benchmark-wide process annotation coverage is 943 of 1,033 evaluation instances (91.3%).
Appendix D.5 defines each requirement-axis composite as 0.33 Outcome + 0.33 Alignment + 0.34 Reasoning Quality, and Overall as the macro-average of the three axis composites. Each reasoning requirement therefore has equal influence on Overall. Requirement-level and component-level scores accompany Overall to make model comparisons explicit across CHEMARENA's five task families and three reasoning requirements.