The Model's Own Taxonomy: How MLLMs Organize Human Cognition Tasks

Human cognitive theories provide taxonomies of abilities involved in perception and reasoning, but how closely these taxonomies characterize multimodal large language model (MLLM) reasoning remains understudied. To examine this correspondence, we cluster examples using model hidden-state representations and compare the resulting clusters with human cognitive categories. Across five MLLMs, these clusters reveal three relationships with the human taxonomy, including close correspondence, subdivision within categories, and regrouping across category boundaries. To test whether these distinctions matter for reasoning, we learn a shared attention-head intervention for each human cognitive category or model cluster and compare its effects on accuracy. Cluster-guided interventions outperform those guided by human cognitive categories and improve held-out test accuracy by 3.75-7.98% over the unmodified models. Together, these findings suggest that human cognitive categories capture part of the organization of model reasoning, while model clusters identify additional distinctions that support effective reasoning interventions.