acceptodds
Under review as a conference paper at ICLR 2027

UniAR: Benchmarking Cross-Modal Evidence-Grounded Anomaly Reasoning for MLLMs

Abstract

Anomaly reasoning is a core perceptual and cognitive ability in humans, who can identify anomalies across distinct modalities and provide corresponding evidence. Nevertheless, there remains a lack of a unified benchmark to rigorously and credibly evaluate such ability in Multimodal Large Language Models (MLLMs) on comprehensive tasks spanning video, image, and text modalities. To this end, we present UniAR, the first cross-modal anomaly reasoning benchmark with full-spectrum tasks and credible evaluation. Built from publicly available source data, UniAR provides over 40,199 high-quality question-answer pairs organized into four shared challenges: dual anomaly discrimination, context understanding, fine-grained anomaly classification, and holistic anomaly analysis, covering coarse-to-fine anomaly reasoning across video, image, and text modalities. To improve credibility, we go beyond standard multiple-choice questions (MCQs) by assessing the evidence MLLMs provide for their answers, measuring whether they genuinely comprehend the anomalous phenomena. We benchmark over 20 latest models, and find that though proficient in semantic understanding and commonsense reasoning, they show clear modality bias in anomaly discrimination and lag substantially behind human performance across all tasks, indicating considerable room for improvement. We hope UniAR can facilitate research on human-like anomaly reasoning for embodied agents and intelligent assistants.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Model, Cross-Modal Anomaly Reasoning, Visual Question Answering, Evidence-based Credible Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8kU5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniAR, a unified benchmark for evaluating the anomaly reasoning capabilities of MLLMs across video, image, and text. Rather than assuming identical anomaly semantics across modalities or requiring multi-modal fusion within each instance, UniAR profiles MLLMs’ anomaly reasoning through a shared set of reasoning tasks and human-normalized performance.

Critical strengths

Evaluating supporting evidence alongside answer correctness is a worthwhile direction for anomaly reasoning. The benchmark covers complementary scenarios, including surveillance events, industrial defects, and textual errors. The paper appropriately clarifies that its contribution is a shared evaluation framework across modalities, rather than multimodal fusion within each instance.

Critical weaknesses
  1. Some comparisons additionally use different model checkpoints across tracks. The analysis should distinguish within-checkpoint comparisons from model-family comparisons, report uncertainty for human-normalized scores, and evaluate models on the same human-assessed subsets.
  2. “Prediction persuasiveness” is also not necessarily equivalent to evidence correctness. The authors should specify an evidence-focused rubric, whether evaluators were blinded to model identity, and whether they were independent of benchmark construction. Sample-level agreement and confidence intervals for rank correlations would be more informative than selected aggregate rank differences.
  3. The overall writing is hard to read, and some paragraphs are overly lengthy. For human readers, such manuscripts require an extremely long time to read, and the words used in the articles also make it difficult for people to accurately understand their meanings.
  4. In terms of engineering work, the question-and-answer pairs constructed in this paper have indeed reached a certain scale. However, from the perspective of the expression of the manuscript, the exploration of the substantive contributions brought by its benchmark is insufficient.
Questions for authors
  1. How does the author ensure the correctness of evidence annotation?
Reviewer_rgmkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UniAR, a benchmark for evaluating anomaly reasoning in videos, images, and text. It contains about 40,000 multiple-choice questions covering anomaly discrimination, context understanding, fine-grained classification, and holistic analysis. In addition to answer accuracy, UniAR evaluates whether models can identify supporting evidence through temporal intervals, bounding boxes, or text spans. Experiments on more than 20 models show substantial gaps from human performance and notable performance differences across modalities.

Critical strengths
  1. The paper studies an important problem that is not well covered by existing benchmarks.
  2. It evaluates anomaly reasoning across video, image, and text within a shared framework.
  3. The benchmark covers multiple reasoning tasks rather than only basic anomaly detection.
  4. The evidence-grounded evaluation is useful because it checks whether a model can support its answer.
  5. The experiments are broad, covering more than 20 open- and closed-source models & human baselines.
Critical weaknesses
  1. The referring setting may leak the label: anomalous samples use local intervals, boxes, or spans, while normal samples use the full input range. A model may therefore predict the answer from the reference format alone.
  2. The human-normalized scores do not fully make the three modalities comparable because their datasets, question types, option counts, and difficulty levels are different.
  3. The video and image tracks are class-imbalanced, but the paper mainly reports accuracy without balanced accuracy, per-class recall, or majority-class baselines.
Questions for authors
  1. How do you prevent label leakage in the referring setting, given that anomalous samples use local references while normal samples use the full input range? Could you report a reference-only baseline and results using matched local regions from normal samples?
  2. Could you report balanced accuracy, per-class recall, and majority-class baselines for the class-imbalanced video and image tracks?
Reviewer_yqngOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UniAR, a unified benchmark for evaluating anomaly reasoning capabilities of MLLMs across video, image, and text. It integrates seven existing datasets and contains 40,199 multiple-choice QA pairs covering four tasks: anomaly discrimination, context understanding, fine-grained anomaly classification, and holistic anomaly analysis. The paper also introduces evidence-grounded evaluation and human-normalized scores to assess prediction reliability and cross-modal performance. Experiments on over 20 MLLMs reveal substantial gaps from human performance, modality-dependent differences, and limitations in evidence-grounded anomaly reasoning.

Critical strengths
  1. The paper constructs a large-scale benchmark with 40,199 QA pairs and fine-grained evidence annotations across three modalities.
  2. The paper provides a comprehensive comparison of over 20 MLLMs.
Critical weaknesses
  1. Insufficient motivation for cross-modal unification. Video-AR and Image-AR focus on abnormal events and industrial visual defects, while Text-AR mainly addresses grammatical, lexical, and factual errors. The paper does not clearly justify why these substantially different tasks should be unified or whether they share meaningful anomaly reasoning capabilities beyond a common QA format.
  2. Limited insights from cross-modal evaluation. The experiments mainly compare model performance across different modalities, without providing deeper insights into general anomaly reasoning capabilities. For example, can foundation models recognize deviations from normality across different modalities? Without addressing such fundamental questions, the value of jointly evaluating reasoning across these distinct tasks remains unclear.
Questions for authors
  1. What is the motivation for unifying Video-AR, Image-AR, and Text-AR, given their substantially different anomaly definitions and required capabilities? What common anomaly reasoning abilities do these tasks share beyond a unified QA format?
  2. Could the authors include comparisons with task-specific MLLM-based methods, such as Triad[1] and IAD-R1[2] for industrial anomaly detection, rather than only evaluating general-purpose MLLMs? This would provide a more comprehensive assessment of model performance within each domain.

[1] Li Y, Yuan S, Wang H, et al. Triad: Empowering lmm-based anomaly detection with expert-guided region-of-interest tokenizer and manufacturing process[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 21917-21926. [2] Li Y, Cao Y, Liu C, et al. Iad-r1: Reinforcing consistent reasoning in industrial anomaly detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(8): 6583-6591.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.