- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents UniAR, a unified benchmark for evaluating the anomaly reasoning capabilities of MLLMs across video, image, and text. Rather than assuming identical anomaly semantics across modalities or requiring multi-modal fusion within each instance, UniAR profiles MLLMs’ anomaly reasoning through a shared set of reasoning tasks and human-normalized performance.
Critical strengths
Evaluating supporting evidence alongside answer correctness is a worthwhile direction for anomaly reasoning. The benchmark covers complementary scenarios, including surveillance events, industrial defects, and textual errors. The paper appropriately clarifies that its contribution is a shared evaluation framework across modalities, rather than multimodal fusion within each instance.
Critical weaknesses
- Some comparisons additionally use different model checkpoints across tracks. The analysis should distinguish within-checkpoint comparisons from model-family comparisons, report uncertainty for human-normalized scores, and evaluate models on the same human-assessed subsets.
- “Prediction persuasiveness” is also not necessarily equivalent to evidence correctness. The authors should specify an evidence-focused rubric, whether evaluators were blinded to model identity, and whether they were independent of benchmark construction. Sample-level agreement and confidence intervals for rank correlations would be more informative than selected aggregate rank differences.
- The overall writing is hard to read, and some paragraphs are overly lengthy. For human readers, such manuscripts require an extremely long time to read, and the words used in the articles also make it difficult for people to accurately understand their meanings.
- In terms of engineering work, the question-and-answer pairs constructed in this paper have indeed reached a certain scale. However, from the perspective of the expression of the manuscript, the exploration of the substantive contributions brought by its benchmark is insufficient.
Questions for authors
- How does the author ensure the correctness of evidence annotation?