acceptodds
Under review as a conference paper at ICLR 2027

HomeACE: A Comprehensive Evaluation Benchmark for Home Anomaly Video Understanding

Abstract

Understanding safety risks in household videos is essential for intelligent systems that support risk prevention and timely response. Achieving this requires a model that can as a forecaster of the household: perceiving subtle risk precursors, inferring causal chains of danger escalation, localizing the temporal window for intervention, and formulating actionable mitigation strategies. However, existing evaluations of multimodal large language models (MLLMs) focus predominantly on general video understanding, with no systematic assessment of the broad range of capabilities required in home settings. To address this gap, we introduce HomeACE, a large-scale Home Anomaly Comprehensive Evaluation dataset comprising more than 14K household videos, including nearly 3K real-world recordings and over 11K generated videos that span normal, risk-only (an emerging risk), and abnormal (a realized anomaly) situations across diverse home scenes. Based on this dataset, we design a systematic evaluation protocol that assesses models along four complementary dimensions: Perception, Cognition, Temporal, and Planning. Extensive experiments reveal substantial limitations in current MLLMs: limited anomaly perception, weak causal reasoning, imprecise temporal localization, and mitigation plans that are often insufficiently grounded in the observed hazards. HomeACE provides a standardized platform for diagnosing these deficiencies and advancing home anomaly understanding toward the vision of prevention-oriented intelligent home systems.

Primary area:
datasets and benchmarks
Keywords:
anomly detection, video understanding, household benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AFKDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HomeACE, a comprehensive benchmark for home anomaly video understanding. The dataset consists of 14,518 real and generated videos, covering three states: normal activities, potential risks, and realized anomalies. The authors further develop an evaluation framework encompassing four dimensions: Perception, Cognition, Temporal Localization, and Planning. Experiments across multiple multimodal large language models reveal limitations in early risk recognition, causal understanding, and temporal localization.

Critical strengths

1.The paper addresses a problem with practical relevance. 2.The benchmark provides a relatively large dataset and a comprehensive evaluation framework.

Critical weaknesses

1.The relationship with existing studies on pre-abnormal understanding is not sufficiently discussed. The prior work Predicting the Unseen: A Novel Dataset for Hidden Intention Localization in Pre-abnormal Analysis (ACM MM 2024) appears to have investigated hidden abnormal intention recognition before abnormal events occur and modeled pre-abnormal cues.

2.The reliability of LLM-based automatic semantic evaluation has not been sufficiently validated. 3.It remains unclear whether the Cognition metric effectively measures visual causal understanding. 4.The high proportion of generated videos may affect the realism and diversity of the benchmark.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Responses to Reviewer AFKD — (1/2)

Q1: Related work on pre-abnormal understanding. We thank the reviewer for pointing out this related work.

We recognize the important contribution of Predicting the Unseen to pre-abnormal understanding: it introduces a hidden abnormal-intention dataset containing 228 videos and primarily focuses on fine-grained behavioral cues preceding anomalous behavior in security scenarios and their temporal localization. HomeACE has a different focus: we study household scenarios, covering multiple risk types including falls, fire, and electrical hazards, as well as Normal, Risk-only, and Abnormal states. The dataset contains 14,518 videos and evaluates multimodal models across four dimensions: perception, cognition, temporal localization, and intervention planning. We have added the citation and discussion to the related-work section and Table 1, with the new text marked in blue.


Q2: Human agreement and judge robustness. We thank the reviewer for the attention to evaluation reliability.

(1) Human–judge agreement. We sampled 300 model responses requiring automatic judging, with equal numbers from real and generated videos. We compute the absolute difference between the human and automatic scores for each output and average these differences over the 300 outputs. On the original 0–5 scale, the mean absolute error is 0.28 points, indicating a relatively small difference. We also compute Spearman rank correlation from the two sets of scores for the 300 responses to measure agreement in their relative ordering, obtaining 0.883. This indicates close agreement in scores and relative ordering, supporting the reliability of the benchmark evaluation.

(2) Judge robustness. We use Mistral-7B [1], InternLM3-8B [2], and GLM-4-9B [3] to evaluate model outputs on the four defined dimensions, and compare overall scores and rankings for Abnormal and Risk-only videos. Overall scores average the real and generated video scores; rankings are in parentheses. The judges assign similar scores and produce largely consistent relative rankings, supporting the robustness of scores across judges and our use of Qwen3-8B as the judge.

Abnormal

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra60.8 (1)58.7 (1)52.0 (1)57.0 (1)
Claude-Opus-558.3 (2)54.9 (2)47.0 (3)53.3 (2)
Gemini-3.1-Pro57.0 (3)50.8 (3)47.1 (2)48.3 (3)
Qwen3.8-27B57.2 (1)52.9 (1)47.5 (1)50.6 (1)
Qwen3.5-9B55.8 (2)52.0 (2)46.8 (2)48.9 (2)
InternVL3.5-8B53.0 (3)46.1 (3)43.1 (3)45.1 (3)
Qwen2.5-VL-7B36.1 (4)30.4 (5)24.2 (4)28.4 (4)
Tarsier2-7B34.5 (5)32.7 (4)23.9 (5)26.9 (5)
VideoLLaMA3-7B29.4 (6)23.6 (6)17.7 (6)21.4 (6)
VideoLLaMA2.1-7B25.6 (7)18.8 (7)12.2 (7)16.6 (7)

Risk-only

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra62.0 (1)58.7 (1)53.2 (1)58.2 (1)
Claude-Opus-553.2 (2)48.7 (2)42.1 (3)48.4 (2)
Gemini-3.1-Pro52.5 (3)46.9 (3)43.9 (2)46.2 (3)
Qwen3.8-27B52.5 (1)46.6 (1)42.3 (1)45.9 (1)
Qwen3.5-9B45.9 (3)40.8 (3)35.9 (3)38.8 (3)
InternVL3.5-8B52.1 (2)43.9 (2)41.7 (2)43.9 (2)
Qwen2.5-VL-7B30.9 (5)25.8 (5)18.1 (5)23.0 (5)
Tarsier2-7B32.1 (4)29.9 (4)20.5 (4)24.3 (4)
VideoLLaMA3-7B21.2 (7)15.8 (7)8.2 (7)13.3 (7)
VideoLLaMA2.1-7B25.6 (6)18.9 (6)10.7 (6)15.7 (6)

References

[1] Jiang, A. Q., et al. Mistral 7B. arXiv:2310.06825, 2023. https://arxiv.org/abs/2310.06825

[2] InternLM Team. InternLM3: An Open-Source Large Language Model, 2025. https://github.com/InternLM/InternLM

[3] Team GLM. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793, 2024. https://arxiv.org/abs/2406.12793


AuthorsOpenReview ↗

Responses to Reviewer AFKD — (2/2)

Q3: Cognition and visual causal understanding. We thank the reviewer for the question.

The causal understanding submetric (Caus) assesses whether a model explains relationships among risk sources, actions, and consequences. Annotators determine reference annotations from the video, and supplemental references are independently reviewed. Qwen3-8B compares model outputs with the reference causal chain under fixed criteria and scores the explanation of risk evolution. This design evaluates video-related causal understanding; shared references and criteria enable consistent, relatively fair comparisons across models.


Q4: Realism and diversity of generated data. We thank the reviewer for the attention to these issues.

(1) Scale, diversity, and collection limitations of real videos. HomeACE contains 2,998 real household videos from five public datasets and multiple online platforms, spanning 10 household scene categories, multiple risk types, and Normal, Risk-only, and Abnormal states. The subset has substantial scale and diversity, but some household risks are difficult to collect in real life, limiting coverage of risk types and specific scenarios.

(2) Coverage and quality control of generated videos. We use generated videos to supplement hard-to-collect risk scenarios and increase diversity. The generated subset contains 11,520 videos across eight household scene categories, eight risk types, and three states (3,840 videos each), generated by Sora 2, Veo 3.1, Grok Video 1.5, and Omini. A hierarchical scene–subject–risk category–specific behavior tree organizes target scenarios, followed by structural checks, multi-perspective risk review, and final screening. All generated videos are manually inspected; failures are removed or regenerated. Annotations and risk intervals for retained videos are corrected against the actual footage to improve consistency and data quality.

Thank you. I believe the additional experiments address my original concerns. I therefore raise my rating from 2 to 3.

AuthorsOpenReview ↗

Thanks for your prompt feedback and support!

We are grateful to hear our feedback has addressed your concerns. We sincerely appreciate your recognition of our contributions and the increased score. Thank you once again for your time and effort in reviewing our paper!

Reviewer_q1zGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HomeACE establishes a comprehensive benchmark for the evaluation of household anomalous video understanding, consisting of 14,518 videos in total, among which 2,998 are real-world videos and 11,520 are synthetic videos. In this paper, household events are categorized into three states: Normal, Risk-only, and Abnormal. Four evaluation dimensions are designed, namely Perception, Cognition, Temporal, and Planning. The authors evaluate 10 multimodal large language models, conduct grouped experiments on real and synthetic videos, and carry out controlled experiments with ground-truth risk temporal windows provided. The primary contribution of this paper lies in the dataset and evaluation framework, rather than a novel anomaly detection algorithm.

Critical strengths

The design of the Risk-only scenario carries practical significance. Conventional anomaly detection tends to focus heavily on events that have already occurred. In contrast, HomeACE distinguishes potential hazard precursors from fully manifested abnormal events. This aligns better with the early risk recognition requirements of home security systems, representing a valuable design choice in this work.

The evaluation dimensions are comprehensive, and the experiments are diagnostic. Beyond risk recognition, the paper separately examines causal reasoning, temporal localization, and mitigation strategies. It also reports results under different data sources and conditions with ground-truth temporal window prompts. The authors acknowledge that a high Planning score does not necessarily mean the model truly understands the video. Therefore, they avoid equating high planning scores with reliable safety decisions. Such discussion is relatively prudent.

Critical weaknesses

The reliability of evaluations for core semantic metrics has not been fully validated. This is the most critical concern in my opinion. The paper heavily relies on Qwen3-8B to score model-generated risk descriptions, causal relationships and mitigation strategies. This judge primarily compares the model’s textual outputs against reference annotations, without re-watching the original videos. Although the authors describe the manual annotation and review pipeline, the work lacks sufficient analyses of independent human evaluation agreement, the correlation between judge outputs and human judgments, and whether model rankings remain stable when the judge is replaced. Accordingly, uncertainty remains regarding whether the Cognition and Planning scores accurately reflect video understanding capabilities.

Generalization to real-world scenarios and the reliability of some experimental results need to be strengthened. Approximately 79% of the videos are synthetic data. While the authors separately report results on real and synthetic splits, there is insufficient analysis of visual biases potentially introduced by different generators, evaluation sample selection, and generalization performance on genuine household scenes. Additionally, there are numerical discrepancies worthy of inspection in Appendix Table 10 (Page 24):

  • For VideoLLaMA3-7B, the full set of sub-metrics under Risk-only and Abnormal settings are identical.
  • For VideoLLaMA2.1-7B under Risk-only, the reported Overall Score is 33.4, yet the average of its three listed dimensions (33.3, 43.0, 43.5) is approximately 39.9. This does not necessarily invalidate all conclusions of the paper, but the authors should check their experimental records and explain these inconsistencies. Another notable observation: the claim that Risk-only is harder than Abnormal does not hold for all models. For instance, in Table 2, InternVL3.5-8B achieves higher state recognition accuracy under the Risk-only setting. The paper should draw a clearer distinction between overall trends and model-specific variations.
Questions for authors

Judge validity: Could the authors provide consistency analysis between independent human scores and scores from Qwen3-8B, and report the stability of model rankings when the judge is replaced? Dataset and evaluation: What is the exact number of videos used for evaluation? How are sample selection, deduplication, and aggregation of real and synthetic data conducted? Can results stratified by generator and risk type together with confidence intervals be provided? Result consistency: Could the authors check and fix the relevant results of VideoLLaMA2.1-7B and VideoLLaMA3-7B in Table 10? Inference protocol: Does the Cognition stage adopt model-predicted risk states or ground-truth states? How are the supplemented reference annotations independently verified? Do different video frame sampling configurations affect the main conclusions?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer q1zG — (1/5)

We first answer the Questions For Authors in order, then address additional issues raised in the Weaknesses, and finally discuss data privacy.

Q1: Human agreement and judge robustness. We thank the reviewer for the attention to evaluation reliability.

(1) Human–judge agreement. We sampled 300 model responses requiring automatic judging, with equal numbers from real and generated videos. We compute the absolute difference between the human and automatic scores for each output and average these differences over the 300 outputs. On the original 0–5 scale, the mean absolute error is 0.28 points, indicating a relatively small difference. We also compute Spearman rank correlation from the two sets of scores for the 300 responses to measure agreement in their relative ordering, obtaining 0.883. This indicates close agreement in scores and relative ordering, supporting the reliability of the benchmark evaluation.

(2) Judge robustness. We use Mistral-7B [1], InternLM3-8B [2], and GLM-4-9B [3] to evaluate model outputs on the four defined dimensions, and compare overall scores and rankings for Abnormal and Risk-only videos. Overall scores average the real and generated video scores; rankings are in parentheses. The judges assign similar scores and produce largely consistent relative rankings, supporting the robustness of scores across judges and our use of Qwen3-8B as the judge.

Abnormal

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra60.8 (1)58.7 (1)52.0 (1)57.0 (1)
Claude-Opus-558.3 (2)54.9 (2)47.0 (3)53.3 (2)
Gemini-3.1-Pro57.0 (3)50.8 (3)47.1 (2)48.3 (3)
Qwen3.8-27B57.2 (1)52.9 (1)47.5 (1)50.6 (1)
Qwen3.5-9B55.8 (2)52.0 (2)46.8 (2)48.9 (2)
InternVL3.5-8B53.0 (3)46.1 (3)43.1 (3)45.1 (3)
Qwen2.5-VL-7B36.1 (4)30.4 (5)24.2 (4)28.4 (4)
Tarsier2-7B34.5 (5)32.7 (4)23.9 (5)26.9 (5)
VideoLLaMA3-7B29.4 (6)23.6 (6)17.7 (6)21.4 (6)
VideoLLaMA2.1-7B25.6 (7)18.8 (7)12.2 (7)16.6 (7)

Risk-only

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra62.0 (1)58.7 (1)53.2 (1)58.2 (1)
Claude-Opus-553.2 (2)48.7 (2)42.1 (3)48.4 (2)
Gemini-3.1-Pro52.5 (3)46.9 (3)43.9 (2)46.2 (3)
Qwen3.8-27B52.5 (1)46.6 (1)42.3 (1)45.9 (1)
Qwen3.5-9B45.9 (3)40.8 (3)35.9 (3)38.8 (3)
InternVL3.5-8B52.1 (2)43.9 (2)41.7 (2)43.9 (2)
Qwen2.5-VL-7B30.9 (5)25.8 (5)18.1 (5)23.0 (5)
Tarsier2-7B32.1 (4)29.9 (4)20.5 (4)24.3 (4)
VideoLLaMA3-7B21.2 (7)15.8 (7)8.2 (7)13.3 (7)
VideoLLaMA2.1-7B25.6 (6)18.9 (6)10.7 (6)15.7 (6)

References

[1] Jiang, A. Q., et al. Mistral 7B. arXiv:2310.06825, 2023. https://arxiv.org/abs/2310.06825

[2] InternLM Team. InternLM3: An Open-Source Large Language Model, 2025. https://github.com/InternLM/InternLM

[3] Team GLM. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793, 2024. https://arxiv.org/abs/2406.12793


Q2: Sample selection, deduplication, and evaluation size. We thank the reviewer for the attention to data processing and evaluation scale.

(1) Sample selection. Real videos are selected from household-related scenes in five public datasets and multiple video platforms. Generated videos are produced from household risk scenarios organized by a hierarchical scene–subject–risk category–specific behavior tree and undergo human verification; videos that do not pass verification are regenerated.

(2) Deduplication.

  • For real videos, we first perform preliminary deduplication using source, title, and description, then manually screen for duplicates.

  • For generated videos, each prompt–generator combination is distinct, and we manually screen for duplicates.

After screening, we aggregate the real and generated videos separately and report the subset results in Appendix Tables 7 and 8; main-text Table 2 averages the two subset results.

(3) Evaluation size. As in the paper, the evaluation dataset contains 2,998 real and 11,520 generated videos, totaling 14,518. The numbers included for individual models may differ slightly because predictions and matching outcomes vary.


AuthorsOpenReview ↗

Responses to Reviewer q1zG — (2/5)

Q3: Per-category analysis of generated videos. We thank the reviewer for suggesting further analysis. We analyze performance by generator and risk type.

(1) By generator. The table reports each evaluated model's mean score on videos from each generator. Performance varies across generators, although scores are broadly comparable in magnitude.

Abnormal

ModelSora 2Veo 3.1GrokOmini
GPT-5.6-Terra59.163.263.363.9
Gemini-3.1-Pro52.656.255.856.3
Claude-Opus-545.754.251.851.5
Qwen3.8-27B52.857.957.957.8
Qwen3.5-9B50.256.856.256.3
InternVL3.5-8B46.554.252.150.8
Qwen2.5-VL-7B-Instruct29.839.847.639.6
Tarsier2-7B32.041.440.743.1
VideoLLaMA2.1-7B19.329.326.326.1
VideoLLaMA3-7B24.443.140.346.2
Average41.349.649.249.1

Risk-only

ModelSora 2Veo 3.1GrokOmini
GPT-5.6-Terra59.365.364.365.9
Gemini-3.1-Pro48.253.451.553.2
Claude-Opus-536.446.543.144.4
Qwen3.8-27B47.253.153.352.6
Qwen3.5-9B38.444.543.343.3
InternVL3.5-8B46.956.052.848.2
Qwen2.5-VL-7B-Instruct28.534.734.530.1
Tarsier2-7B28.034.428.630.7
VideoLLaMA2.1-7B18.925.619.924.2
VideoLLaMA3-7B16.324.318.624.2
Average36.843.841.041.7

(2) By risk type. The table reports the 10 models' scores and their average by risk type. As on real videos, models score relatively highly on fire risks. On generated videos, falls score relatively highly under Abnormal and electrical hazards under Risk-only, whereas poisoning scores relatively low in both states. This may reflect differences in visual cue salience and scene complexity.

Abnormal

Risk typeGPTGeminiOpusQ3.8Q3.5InternQ2.5TarsierVL2.1VL3Avg
Fall risk64.657.853.962.059.457.350.648.828.750.653.4
Heat and fire65.456.858.658.557.154.950.445.830.451.853.0
Electrical hazard66.657.754.458.757.055.146.745.330.449.252.1
Interpersonal conflict62.354.652.957.655.653.848.544.429.253.651.3
Collision injury59.553.244.554.454.052.938.742.125.343.346.8
Sharp objects61.955.749.355.852.449.931.635.722.625.244.0
Property security60.854.751.457.958.945.233.526.120.816.442.6
Poisoning61.154.048.152.249.744.326.832.422.129.742.0

Risk-only

Risk typeGPTGeminiOpusQ3.8Q3.5InternQ2.5TarsierVL2.1VL3Avg
Electrical hazard68.059.150.356.751.858.037.334.324.224.546.4
Heat and fire65.153.648.955.446.254.034.533.222.325.843.9
Fall risk65.754.246.154.840.659.330.834.921.823.943.2
Interpersonal conflict66.652.539.056.146.452.738.725.627.624.943.0
Property security62.448.639.352.954.551.745.030.923.117.342.6
Sharp objects67.956.344.054.738.451.827.529.320.419.140.9
Poisoning61.746.941.643.834.742.424.829.420.117.336.3
Collision injury57.345.240.344.232.446.524.529.720.719.236.0

The columns report scores for the 10 evaluated models; Avg is their mean. Q3.8, Q3.5, and Q2.5 denote the corresponding Qwen models; Intern denotes InternVL3.5-8B; VL2.1 and VL3 denote the corresponding VideoLLaMA models.

(3) Confidence intervals. We calculate 95% confidence intervals for mean scores by generator and risk type using 5,000 bootstrap samples with replacement at the risk-behavior level. The results are shown below.

CategoryAbnormalRisk-only
Generator
Sora 240.4–42.135.9–37.8
Veo 3.149.1–50.143.2–44.3
Grok48.8–49.640.4–41.6
Omini48.5–49.740.9–42.5
Risk type
Fall risk52.5–54.241.9–44.5
Interpersonal conflict50.6–52.041.5–44.6
Property security41.8–43.341.9–43.3
Collision injury45.6–47.934.5–37.4
Heat and fire52.2–53.742.7–45.1
Poisoning40.9–43.234.8–37.8
Electrical hazard51.3–53.045.1–47.7
Sharp objects42.9–45.139.6–42.3

AuthorsOpenReview ↗

Responses to Reviewer q1zG — (3/5)

Q4: Consistency of Table 10. We thank the reviewer for identifying the inconsistency. Checking the original results revealed that two Abnormal results were mistakenly entered in Risk-only rows. We corrected Appendix Table 10 using the original records and marked the changes in blue. The errors do not affect experimental results in the main-text tables.


Q5: Risk status used in cognition. We use single-pass inference: given the video and task prompt, the model itself outputs the risk status and cognition fields in the same pass. Ground-truth risk status is not provided as input, so it cannot leak into the prediction.


Q6: Validation of supplemental reference annotations. The supplemental reference annotations were also independently reviewed by annotators.


Q7: Frame sampling. We thank the reviewer for the question.

(1) Setup: We add comparisons with 16- and 8-frame sampling and the main-table results. The table reports overall scores for ten models on all Abnormal and Risk-only videos. Except for VideoLLaMA2.1-7B, which uses 16 frames to match its training configuration, the models use 32, 16, and 8 frames, respectively.

(2) Results: Most models perform better with 16 than 32 frames, while scores generally decline with 8 frames. Moderately reducing the number of frames may reduce redundant information, but too few frames may omit key risk cues; sensitivity varies across models.

Abnormal

Model32 frames16 frames8 frames
GPT-5.6-Terra60.864.162.3
Claude-Opus-558.361.248.8
Gemini-3.1-Pro57.061.761.3
Qwen3.8-27B57.259.457.7
Qwen3.5-9B55.858.957.6
InternVL3.5-8B53.053.953.5
Qwen2.5-VL-7B-Instruct36.137.732.1
Tarsier2-7B34.531.534.4
VideoLLaMA2.1-7B–25.6–
VideoLLaMA3-7B29.417.024.7

Risk-only

Model32 frames16 frames8 frames
GPT-5.6-Terra62.063.863.2
Claude-Opus-553.254.140.1
Gemini-3.1-Pro52.556.355.8
Qwen3.8-27B52.552.350.4
Qwen3.5-9B45.947.745.6
InternVL3.5-8B52.154.253.5
Qwen2.5-VL-7B-Instruct30.929.925.6
Tarsier2-7B32.128.530.9
VideoLLaMA2.1-7B–25.6–
VideoLLaMA3-7B21.211.717.0

Reviewer_SRQ9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HomeACE, a large-scale benchmark for home anomaly and risk understanding. It contains 14,518 household videos (2,998 real-world + 11,520 generated) organized by a hierarchical risk tree, with three risk states: normal, risk-only (visible precursors without realized consequences), and abnormal (realized incidents). The paper also proposes a four-dimensional evaluation protocol — Perception, Cognition, Temporal, and Planning — and evaluates ten proprietary/open-source MLLMs. Key findings: models recognize realized incidents far better than emerging risks, factual recognition does not imply causal understanding, and temporal localization is the main bottleneck that propagates to other capabilities.

Critical strengths
  1. Meaningful gap. Existing anomaly benchmarks are surveillance-centric and do not model home-specific risk progression; separating risk-only from realized anomalies is a useful formulation.
  2. Scale and coverage. 14K+ videos across diverse home scenes, subjects, and risk categories; the generated subset is balanced across risk states (3,840 per state).
  3. Structured construction pipeline. Hierarchical risk tree, root-to-leaf prompt expansion, multi-perspective risk review (Table 4), and human verification are documented with prompts and examples in the appendix.
Critical weaknesses
  1. Synthetic data dominates (79% of videos). The real-world subset is only 2,998 videos, and the generated subset is produced by Sora 2, Veo 3.1, Grok Video 1.5, and Omini. The paper does not analyze whether model rankings or conclusions differ between real and generated subsets beyond reporting subset-specific tables. A real-vs-generated gap analysis in the main text is needed.
  2. LLM-assisted annotation and LLM judge create circularity. Annotations are assisted by five MLLMs and completed by GPT-5.5; semantic outputs are scored by Qwen3-8B. The paper does not report human–judge agreement, judge robustness, or whether evaluated models were also used in annotation. This is a serious validity concern for a benchmark paper.
Questions for authors
  1. What is the human–judge agreement between Qwen3-8B scores and expert ratings on a held-out sample? How sensitive are the rankings to the judge model?
  2. Were any of the evaluated MLLMs (e.g., Qwen3.5-9B, InternVL3.5-8B, VideoLLaMA3) also used in annotation or consistency screening? If so, how do you avoid favoring them?
  3. What are the inter-annotator agreement numbers for risk state, risk type, and temporal intervals?
  4. How do conclusions differ between the real-world and generated subsets? Can you provide a direct real-vs-generated comparison in the main text?
3 comments
AuthorsOpenReview ↗

Responses to Reviewer SRQ9 — (1/2)

Q1: Human agreement and judge robustness. We thank the reviewer for the attention to evaluation reliability.

(1) Human–judge agreement. We sampled 300 model responses requiring automatic judging, with equal numbers from real and generated videos. We compute the absolute difference between the human and automatic scores for each output and average these differences over the 300 outputs. On the original 0–5 scale, the mean absolute error is 0.28 points, indicating a relatively small difference. We also compute Spearman rank correlation from the two sets of scores for the 300 responses to measure agreement in their relative ordering, obtaining 0.883. This indicates close agreement in scores and relative ordering, supporting the reliability of the benchmark evaluation.

(2) Judge robustness. We use Mistral-7B [1], InternLM3-8B [2], and GLM-4-9B [3] to evaluate model outputs on the four defined dimensions, and compare overall scores and rankings for Abnormal and Risk-only videos. Overall scores average the real and generated video scores; rankings are in parentheses. The judges assign similar scores and produce largely consistent relative rankings, supporting the robustness of scores across judges and our use of Qwen3-8B as the judge.

Abnormal

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra60.8 (1)58.7 (1)52.0 (1)57.0 (1)
Claude-Opus-558.3 (2)54.9 (2)47.0 (3)53.3 (2)
Gemini-3.1-Pro57.0 (3)50.8 (3)47.1 (2)48.3 (3)
Qwen3.8-27B57.2 (1)52.9 (1)47.5 (1)50.6 (1)
Qwen3.5-9B55.8 (2)52.0 (2)46.8 (2)48.9 (2)
InternVL3.5-8B53.0 (3)46.1 (3)43.1 (3)45.1 (3)
Qwen2.5-VL-7B36.1 (4)30.4 (5)24.2 (4)28.4 (4)
Tarsier2-7B34.5 (5)32.7 (4)23.9 (5)26.9 (5)
VideoLLaMA3-7B29.4 (6)23.6 (6)17.7 (6)21.4 (6)
VideoLLaMA2.1-7B25.6 (7)18.8 (7)12.2 (7)16.6 (7)

Risk-only

ModelQwen3-8BMistral-7BInternLM3-8BGLM-4-9B
GPT-5.6-Terra62.0 (1)58.7 (1)53.2 (1)58.2 (1)
Claude-Opus-553.2 (2)48.7 (2)42.1 (3)48.4 (2)
Gemini-3.1-Pro52.5 (3)46.9 (3)43.9 (2)46.2 (3)
Qwen3.8-27B52.5 (1)46.6 (1)42.3 (1)45.9 (1)
Qwen3.5-9B45.9 (3)40.8 (3)35.9 (3)38.8 (3)
InternVL3.5-8B52.1 (2)43.9 (2)41.7 (2)43.9 (2)
Qwen2.5-VL-7B30.9 (5)25.8 (5)18.1 (5)23.0 (5)
Tarsier2-7B32.1 (4)29.9 (4)20.5 (4)24.3 (4)
VideoLLaMA3-7B21.2 (7)15.8 (7)8.2 (7)13.3 (7)
VideoLLaMA2.1-7B25.6 (6)18.9 (6)10.7 (6)15.7 (6)

References

[1] Jiang, A. Q., et al. Mistral 7B. arXiv:2310.06825, 2023. https://arxiv.org/abs/2310.06825

[2] InternLM Team. InternLM3: An Open-Source Large Language Model, 2025. https://github.com/InternLM/InternLM

[3] Team GLM. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793, 2024. https://arxiv.org/abs/2406.12793


Q2: Overlap between annotation and evaluated models.

We thank the reviewer for this question. Of the five models assisting annotation, Qwen3.5-9B, InternVL3.5-8B, and VideoLLaMA3 were also evaluated. We discuss the annotation process and evaluation results to explain why the potential bias introduced by this overlap is limited.

(1) Human annotation and review.

  • For real videos, the assisting models provide only candidate information, such as descriptions. Annotators watch each video, combine and rephrase information from multiple models to produce an annotation, and other annotators review it. For fields not covered by the initial annotations, GPT-5.5 generates candidates from existing fields; these supplemental reference annotations are also independently reviewed and revised by annotators. The final references are therefore determined by annotators based on the video rather than taken directly from any single model, reducing dependence on that model's output style.

  • For generated videos, model scores of prompt–video consistency are used only for preliminary quality screening; annotators watch the videos and decide whether to retain them.

(2) Evaluation process and results. Annotation assistance and evaluation use different task instructions: during annotation, models provide candidate descriptions for annotators; during evaluation, models respond to the video and task prompts across Perception, Cognition, Temporal, and Planning. The evaluated models that assisted annotation do not rank first overall. For example, in the judge-robustness experiment above, Qwen3.5-9B, InternVL3.5-8B, and VideoLLaMA3 all score below Qwen3.8-27B in their group. These results do not show a clear ranking advantage from assisting annotation.

AuthorsOpenReview ↗

Responses to Reviewer SRQ9 — (2/2)

Q3: Inter-annotator agreement. We thank the reviewer for the question.

We randomly sampled 30% of the videos for independent annotation by two annotators, who could not see each other's annotations. We calculated inter-annotator agreement using these independent annotations. Agreement was 99.0% when distinguishing normal from risk-related videos and 70.9% when distinguishing Risk-only from Abnormal states; agreement on risk type was 91.8%, and mean temporal interval IoU was 0.88. These results indicate high agreement on whether a video is risk-related, while distinguishing potential risk from an actual anomaly is more challenging. Four computer science PhD students divided the initial annotation work; two additional annotators reviewed the results and resolved disagreements through discussion, particularly over the distinction between Risk-only and Abnormal, to produce the final annotations.


Q4: Comparison of real and generated videos. We thank the reviewer for the attention to dataset composition and the results on real and generated videos.

(1) Scale of real videos. Real household risk and anomaly events are infrequent, and privacy concerns make the associated videos difficult to collect. Nevertheless, HomeACE contains 2,998 real videos across Normal (1,085), Risk-only (818), and Abnormal (1,095) states. The real subset is larger than the full video collections of most anomaly datasets in Table 1, such as UCF-Crime (1,900) and SmartHome-Bench (1,203).

(2) Results on real and generated subsets. Appendix Tables 6–8 report results for both video types. We have added the following comparison to the main text, with new content shown in blue.

  • Similarities. The same six models rank highest in overall score on both subsets under both Abnormal and Risk-only settings. Most models achieve lower status recognition scores on Risk-only than on Abnormal videos, suggesting that anticipating risk from precursors is more challenging than recognizing an actual risk event.

  • Differences. Under both settings, most models achieve higher perception scores but lower temporal localization scores on generated videos. Generated videos may have simpler visual content, making risk cues easier to recognize, while shorter reference event intervals make localization scores more sensitive to boundary errors. Rankings also differ: under Abnormal, Claude-Opus-5 ranks first on the real subset, whereas GPT-5.6-Terra ranks first on the generated subset.

AuthorsOpenReview ↗

Follow-up on Rebuttal Feedback for Submission #1685

Dear Reviewer SRQ9,

We sincerely thank you for your positive feedback and thoughtful comments on our submission #1685. We truly appreciate your time and effort in reviewing our work.

As the discussion period draws to a close, we would like to kindly follow up on our rebuttal responses. We would greatly appreciate it if you could share any further thoughts on whether our clarifications have adequately addressed your questions and concerns. Should any concerns remain, we would be happy to provide additional clarification.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.