acceptodds
Withdrawn from ICLR 2027

Ontology-Harnessing Multi-Agent System for Abstractive Visual Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-modal large language models (MLLMs) have made rapid progress in visual understanding, yet abstractive visual representations remain challenging because their interpretation depends on recovering entities, relations, and structural constraints beyond visual appearance alone. In this paper, we propose the Ontology-Harnessing Multi-Agent system (OHM), a training-free framework that decomposes abstractive visual understanding into perception, validation, and task-specific reasoning around an explicit symbolic graph. An ontology provides shared semantic and structural knowledge for intermediate validation and disambiguation, while gradually incorporating reliable observations during inference to support subsequent reasoning. Extensive experiments across multiple tasks and model backbones demonstrate consistent improvements over direct inference, while further revealing the complementary roles of reliable structure recovery and effective downstream reasoning.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multimodal Large Language Models, Abstractive Visual Understanding, Ontology, Multi-Agent Systems, Visual Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_d9yuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free framework which decomposes abstractive visual understanding into perception, validation, and task-specific reasoning around an explicit symbolic graph. This framework improves the reasoning performance by incorporating structural knowledge. The experiments demonstrate the effectiveness of the framework.

Critical strengths

S1. This framework is clearly described.

S2. The experiments validate the framework by using different LLMs.

Critical weaknesses

W1. The importance of explicit structure for agents is not sufficiently addressed in the introduction.

W2. The proposed framework seems technically trivial.

W3. The baseline comparisons are somewhat insufficient. As shown in Table 1, the method is only compared against the “direct” baseline.

Questions for authors

Q1. Why is structured knowledge important for agents? What challenges arise when introducing structured knowledge? Could the authors add more discussion in the introduction?

Q2. What are the technical challenges addressed by the proposed framework? How does the framework's innovation differ from existing graph-based multi-agent approaches [1, 2]?

Q3. Could the authors please provide additional and stronger baseline comparisons, e.g., [1], [2], and other recent harnessing frameworks?


[1] Chen et al. Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System. 2025

[2] Zhang et al. AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering. 2026

Reviewer_bJwnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OHM, a training-free multi-agent framework for abstractive visual understanding. OHM decomposes inference into visual graph reconstruction, ontology-guided validation, and task-specific reasoning for counting, anomaly detection, and completion. Experiments on M3STR and AI2D show that OHM can improve over direct inference across several models.

Critical strengths
  1. OHM provides a carefully designed solution for visual graph-related problems.
  2. The effectiveness of the proposed framework is validated across multiple open-source and proprietary models.
Critical weaknesses
  1. Several important methodological details are unclear.
  • Section 3.2 does not specify how many seed examples are used or how they are selected. Although the authors state that the seed data are outside the target evaluation set, it remains unclear whether they contain entities or relations that also appear in the test data.
  • The paper also does not explain whether the semantic definitions, positive/negative examples, and boundary rules are manually written or generated by an LLM.
  1. The generalizability of the proposed framework is limited.
  • The orchestration module considers only three predefined task families: counting, detection, and completion. It is unclear how OHM would support tasks outside these categories.
  • More fundamentally, the method assumes that abstractive visual inputs can be faithfully represented as entity–relation graphs, but the paper does not establish whether this assumption holds for broader abstractive visual-understanding problems (and even other visual problems).
  1. The central contribution of the ontology is not adequately validated. The paper does not provide a clean quantitative comparison between OHM with and without ontology. The paper should also analyze which tasks and which agents benefit most from ontology information.

  2. Online ontology refinement introduces potential interactions among test examples. Information extracted from earlier evaluation examples is used to update the ontology for subsequent examples. Therefore, it is unclear whether the reported performance depends on the evaluation order or test-time information sharing.

  3. The experimental comparisons are not sufficiently convincing.

  • The paper does not compare against sufficiently strong individual models, such as GPT-5.6, Claude 5, or Qwen3.8.
  • It also lacks comparisons with other general-purpose agentic or harness frameworks and closely related structured visual-reasoning systems.
  • Most substantive experiments are conducted on M3STR, while AI2D provides only limited transfer validation, raising concerns that the method may be tailored to the structure and task definitions of M3STR.
  • The improvements are inconsistent across tasks and model backbones. OHM performs strongly on counting, but produces limited or negative gains in several anomaly-detection and completion settings. In many cases, the improvement depends on replacing the original reasoner.
  • The additional inference cost is not evaluated.
  1. Some references appear incomplete or potentially incorrect. For example, the citation “Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Min Zhang, Wen Zhang, and Huajun Chen. Abstractive visual understanding of multi-modal structured knowledge: A new perspective for MLLM evaluation, 2025” does not provide a conference, journal, arXiv identifier, or other verifiable publication source. The authors should carefully check and complete the bibliography.
Questions for authors

Most questionable points are raised in the weakness part.

Reviewer_XX9eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ontology-Harnessing Multi-Agent system (OHM), a training-free framework that decomposes abstractive visual understanding into perception, validation, and task-specific reasoning around an explicit symbolic graph.

The paper evaluates OHM primarily on M3STR, comparing against a Direct inference baseline across several MLLM backbones, including Qwen2.5-VL, Qwen3-VL, InternVL3.5, MiniCPM-V, DeepSeek-Flash, Gemini-3.1-Flash, and GPT-4o-mini. On M3STR, OHM achieves substantial improvements in counting and anomaly detection settings, while improvements in completion are more limited. The paper additionally compares chain-of-thought prompting and a simplified one-step perception-to-reasoning pipeline (1-Step P→R) on several selected tasks of M3STR on Qwen3-VL-4B and DeepSeek-Flash. The paper also experiments if performance of OHM applies to AI2D.

Critical strengths

S1. Broad evaluation across different MLLM backbones

The paper evaluates OHM across a wide range of MLLM backbones on M3STR, including Qwen2.5-VL, Qwen3-VL, InternVL3.5, MiniCPM-V, DeepSeek-Flash, Gemini-3.1-Flash, and GPT-4o-mini. This is useful for understanding how the proposed framework performs across different models, rather than relying on results from a single backbone.

Critical weaknesses

W1. Limited evaluation beyond structured knowledge graphs

(a) Most of the experiments are conducted on M3STR, which consists of synthetically rendered knowledge graphs. The tasks of M3STR seem to be particularly suited to the proposed method, where counting can be done directly on the extracted graph and anomaly detection/ completion may rely heavily on entity types and relation compatibility. It is therefore unclear whether the improvements would generalize to more diverse visual reasoning tasks.

(b) The authors do evaluate on AI2D, but the comparisons in Figure 5 use DeepSeek-Flash as the reasoning model for OHM, while the Direct baseline uses Qwen2.5-VL-7B or Qwen3-VL-4B only. This makes it difficult to separate the benefit of the method from that of using a different reasoning model.

(c) It would be helpful to include same-model comparisons on AI2D and evaluations on more diverse visual reasoning benchmarks.

W2. Lack of inference-budget-matched comparisons

(a) OHM uses multiple model calls, while the Direct inference baseline only uses a single model call. These may result in significant difference in token count, making it unclear if the gains come from OHM’s method design or just more inference compute. It would be helpful to report the inference compute of OHM vs Direct inference.

(b) Although the authors compare against CoT and 1-Step P→R in Figure 4, there is no comparison with matched inference budgets (eg token usage or number of model calls). Additionally, these results are only shown for a subset of tasks of M3STR and 2 models. It would be helpful to run inference-compute-matched baselines vs OHM.

Questions for authors

Q1. Could the authors provide same-model comparisons between Direct and OHM on AI2D, and did the authors consider evaluating on more diverse visual reasoning benchmarks to better demonstrate generalization of OHM beyond M3STR?

Q2. Could the authors report the inference costs of OHM (eg token usage and number of model calls) and provide results for inference-budget-matched baselines?

Reviewer_fL4pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

To improve MLLM reasoning on abstractive visual representations such as visual knowledge graphs and scientific diagrams, the authors introduce OHM, a training-free multi-agent system. OHM first uses perception agents to convert an input image into an explicit symbolic entity-relation graph. A validation module then checks this graph against a dynamically updated domain ontology to produce a validated graph and soft compatibility evidence. Finally, a task-specific reasoning agent uses only the validated symbolic state and compatibility signals to answer counting, anomaly detection, or completion queries across M3STR and AI2D.

Critical strengths
  1. The disentanglement of visual perception and downstream reasoning is insightful. By swapping perception and reasoning backbones independently, the paper clearly shows that perception quality governs symbolic graph fidelity for counting, while reasoning capacity bottlenecks anomaly detection and completion.
  2. Externalizing an explicit symbolic graph brings large gains on visual counting tasks. Converting visual elements into explicit node and edge sets before counting boosts open-source VLMs by 20 to 48 percentage points on Count-E and Count-R.
Critical weaknesses
  1. The overall motivation is unconvincing, as the method is tailored to a narrow toy scenario and remains unproven on frontier VLMs. Parsing images into rigid entity-relation graphs with predefined ontologies is largely restricted to synthetic node-link diagrams rather than general visual understanding, and the pipeline is only tested on small or lightweight models (4B–8B and Flash/mini variants) without surpassing strong mainstream VLMs.
  2. The benchmark selection and evaluation protocol are overly restricted and fail to demonstrate generalization. Nearly all experiments rely on M3STR—where the orchestrator and reasoning prompts are custom-built for its three fixed sub-tasks—while the only external benchmark (AI2D) is tested on just two small open-source models without broader evaluation on standard chart, document, or scientific reasoning benchmarks.
  3. Direct ablations for the core ontology component and its dynamic refinement are missing. The paper never evaluates OHM without the ontology or compares a static seed ontology against dynamic updates, leaving the namesake contribution unproven.
  4. The mathematical formulation is a black-box placeholder and inference cost is unexamined. Section 3 only writes abstract function notations without defining how compatibility evidence or ontology updates are computed, and the paper omits token/latency overhead as well as recent multi-agent baselines.
Questions for authors

See the weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.