ChemArena: Evaluating Task-Specific Reasoning Requirements in Multimodal Chemical Reasoning

We present ChemArena, a task-aware benchmark for evaluating observable reasoning requirements in multimodal chemical reasoning. It contains 1,033 instances spanning five representative chemical task families, organizing three observable reasoning requirements: Evidence Utilization, Constraint Adherence, and Cross-Turn State Maintenance. To evaluate these task-specific requirements, ChemArena introduces a requirement-aware evaluation protocol that integrates task-specific outcome verification, reference-guided process alignment, and context-enriched reasoning assessment. Experiments on 16 representative MLLMs demonstrate that models with similar task-level performance can exhibit substantially different reasoning behaviors across chemical scenarios. ChemArena provides a systematic framework for analyzing multimodal chemical reasoning beyond accuracy-based evaluation and enables fine-grained understanding of model capabilities and limitations.