- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HomeACE, a comprehensive benchmark for home anomaly video understanding. The dataset consists of 14,518 real and generated videos, covering three states: normal activities, potential risks, and realized anomalies. The authors further develop an evaluation framework encompassing four dimensions: Perception, Cognition, Temporal Localization, and Planning. Experiments across multiple multimodal large language models reveal limitations in early risk recognition, causal understanding, and temporal localization.
Critical strengths
1.The paper addresses a problem with practical relevance. 2.The benchmark provides a relatively large dataset and a comprehensive evaluation framework.
Critical weaknesses
1.The relationship with existing studies on pre-abnormal understanding is not sufficiently discussed. The prior work Predicting the Unseen: A Novel Dataset for Hidden Intention Localization in Pre-abnormal Analysis (ACM MM 2024) appears to have investigated hidden abnormal intention recognition before abnormal events occur and modeled pre-abnormal cues.
2.The reliability of LLM-based automatic semantic evaluation has not been sufficiently validated. 3.It remains unclear whether the Cognition metric effectively measures visual causal understanding. 4.The high proportion of generated videos may affect the realism and diversity of the benchmark.
Questions for authors
see weakness
4 comments
Responses to Reviewer AFKD — (2/2)
Q3: Cognition and visual causal understanding. We thank the reviewer for the question.
The causal understanding submetric (Caus) assesses whether a model explains relationships among risk sources, actions, and consequences. Annotators determine reference annotations from the video, and supplemental references are independently reviewed. Qwen3-8B compares model outputs with the reference causal chain under fixed criteria and scores the explanation of risk evolution. This design evaluates video-related causal understanding; shared references and criteria enable consistent, relatively fair comparisons across models.
Q4: Realism and diversity of generated data. We thank the reviewer for the attention to these issues.
(1) Scale, diversity, and collection limitations of real videos. HomeACE contains 2,998 real household videos from five public datasets and multiple online platforms, spanning 10 household scene categories, multiple risk types, and Normal, Risk-only, and Abnormal states. The subset has substantial scale and diversity, but some household risks are difficult to collect in real life, limiting coverage of risk types and specific scenarios.
(2) Coverage and quality control of generated videos. We use generated videos to supplement hard-to-collect risk scenarios and increase diversity. The generated subset contains 11,520 videos across eight household scene categories, eight risk types, and three states (3,840 videos each), generated by Sora 2, Veo 3.1, Grok Video 1.5, and Omini. A hierarchical scene–subject–risk category–specific behavior tree organizes target scenarios, followed by structural checks, multi-perspective risk review, and final screening. All generated videos are manually inspected; failures are removed or regenerated. Annotations and risk intervals for retained videos are corrected against the actual footage to improve consistency and data quality.
Thank you. I believe the additional experiments address my original concerns. I therefore raise my rating from 2 to 3.
Thanks for your prompt feedback and support!
We are grateful to hear our feedback has addressed your concerns. We sincerely appreciate your recognition of our contributions and the increased score. Thank you once again for your time and effort in reviewing our paper!
Responses to Reviewer AFKD — (1/2)
Q1: Related work on pre-abnormal understanding. We thank the reviewer for pointing out this related work.
We recognize the important contribution of Predicting the Unseen to pre-abnormal understanding: it introduces a hidden abnormal-intention dataset containing 228 videos and primarily focuses on fine-grained behavioral cues preceding anomalous behavior in security scenarios and their temporal localization. HomeACE has a different focus: we study household scenarios, covering multiple risk types including falls, fire, and electrical hazards, as well as Normal, Risk-only, and Abnormal states. The dataset contains 14,518 videos and evaluates multimodal models across four dimensions: perception, cognition, temporal localization, and intervention planning. We have added the citation and discussion to the related-work section and Table 1, with the new text marked in blue.
Q2: Human agreement and judge robustness. We thank the reviewer for the attention to evaluation reliability.
(1) Human–judge agreement. We sampled 300 model responses requiring automatic judging, with equal numbers from real and generated videos. We compute the absolute difference between the human and automatic scores for each output and average these differences over the 300 outputs. On the original 0–5 scale, the mean absolute error is 0.28 points, indicating a relatively small difference. We also compute Spearman rank correlation from the two sets of scores for the 300 responses to measure agreement in their relative ordering, obtaining 0.883. This indicates close agreement in scores and relative ordering, supporting the reliability of the benchmark evaluation.
(2) Judge robustness. We use Mistral-7B [1], InternLM3-8B [2], and GLM-4-9B [3] to evaluate model outputs on the four defined dimensions, and compare overall scores and rankings for Abnormal and Risk-only videos. Overall scores average the real and generated video scores; rankings are in parentheses. The judges assign similar scores and produce largely consistent relative rankings, supporting the robustness of scores across judges and our use of Qwen3-8B as the judge.
Abnormal
Risk-only
References
[1] Jiang, A. Q., et al. Mistral 7B. arXiv:2310.06825, 2023. https://arxiv.org/abs/2310.06825
[2] InternLM Team. InternLM3: An Open-Source Large Language Model, 2025. https://github.com/InternLM/InternLM
[3] Team GLM. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793, 2024. https://arxiv.org/abs/2406.12793