Ontology-Harnessing Multi-Agent System for Abstractive Visual Understanding

Multi-modal large language models (MLLMs) have made rapid progress in visual understanding, yet abstractive visual representations remain challenging because their interpretation depends on recovering entities, relations, and structural constraints beyond visual appearance alone. In this paper, we propose the Ontology-Harnessing Multi-Agent system (OHM), a training-free framework that decomposes abstractive visual understanding into perception, validation, and task-specific reasoning around an explicit symbolic graph. An ontology provides shared semantic and structural knowledge for intermediate validation and disambiguation, while gradually incorporating reliable observations during inference to support subsequent reasoning. Extensive experiments across multiple tasks and model backbones demonstrate consistent improvements over direct inference, while further revealing the complementary roles of reliable structure recovery and effective downstream reasoning.