- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Video-HOCA, a diagnostic benchmark for how video-LLMs judge and explain everyday physical anomalies, organized by a two-level Ontological/Causal taxonomy with finer categories below. On a pool of 630 single-anomaly videos, most of them from IPV-Bench, it builds four tasks (binary plausibility against real Panda-70M clips, nine-way category attribution, four-option recognition, and open-ended explanation scored by an LLM judge), plus a control set in which normal and anomalous videos come from the same generator. Across 20 models, recognition is much stronger than attribution and no model reaches the human reference; on the generator-matched control, every open-weight model loses accuracy and the best model on the original split becomes the worst. Further analyses cover frame shuffling, Thinking, frame count, and judge and rubric robustness, some of them on only one to four models.
Critical strengths
Solid Experiments On a same-generator control (Tables 3–4), every open-weight model loses accuracy, the ranking changes substantially, and the top model on the original split becomes the worst, though all models stay above chance. This shows that real-vs-generated benchmarks partly reward source detection, which is useful for the whole community.
Reliability checks and reproducibility are strong. A second judge and a human check agree closely with the main judge, scores are insensitive to response length and rubric wording, and the judge prompt and z-score reference statistics are published (A.14–A.15). Code and data are available, although raw model outputs, the static/dynamic split file and the per-clip license manifest are not released.
Critical weaknesses
- Main concern is human label. The human baseline is not explained at all: the number of people, the number of items and their professional background are not given, nor is it said whether these people are the ones who wrote the reference answers.
- Several cells of GLM-4.6V-106B and Qwen3.5-35B are below chance, but their Task IV scores are normal, which suggests the cause is more likely an answer-extraction/format problem than an ability problem; the paper does not report the invalid/unparsed rate.
Additional (minor) issues
1 The related-work survey is not thorough: the most recent paper compared in Table 1 was released in May 2025, and there is no comparison with more recent papers. PhyWorldBench [1] (ICLR 2026 Oral) and Order from Chaos [2] (TMLR 2026), TRAVL [3] and PhyGround [4]. Many other papers can also be found. 2. The main text has only two tables. The generator-matched control (Tables 3–4), arguably the paper's main result, appears only in the appendix; it should be in the main text. 3. The paper does not describe how the human annotators were recruited, whether they were compensated, or whether informed consent was obtained (Ethics Statement).
References:
[1] Jing Gu, Xian Liu, Yu Zeng, et al. PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models. In ICLR, 2026 (Oral). https://openreview.net/forum?id=rlZeILv3fm
[2] Meng Cao, Haoran Tang, Haoze Zhao, et al. Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos. Transactions on Machine Learning Research (TMLR), 2026. https://openreview.net/forum?id=Oe5TdpPv1b
[3] Saman Motamed, Minghao Chen, Luc Van Gool, and Iro Laina. TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility. arXiv:2510.07550, 2025. https://arxiv.org/abs/2510.07550
[4] Juyi Lin, Arash Akbari, Yumei He, et al. PhyGround: Benchmarking Physical Reasoning in Generative World Models. arXiv:2605.10806, 2026. https://arxiv.org/abs/2605.10806
Questions for authors
- Annotation agreement. What is the protocol of the human baseline in Table2 ?
- Three annotators re-labeled 120 stratified anomalous videos using the frozen guideline. Are these 3 annotators independent of the writers of the labels and reference answers? If these 3 people are the authors, or the group who wrote the guideline and did the data review, they received the same training, so κ would be inflated and cannot represent the agreement of external annotators.
- Compared with the recent work in [1–4], what are your advantages?
- Report the invalid/unparsed rate of Table 2.
3 comments
Thank you for your comment! I confirm my acceptance 3 rating. It is a good paper.
Thank You for Your Support and Feedback
Thank you for your positive assessment and constructive suggestions. Your feedback on the human-baseline protocol, related work, and bringing the control results into the main text helped strengthen the paper.
Response to Reviewer RB9G
We thank Reviewer RB9G for the encouraging assessment—highlighting our generator-matched control and reliability analyses—and for helpful suggestions on human-protocol details, option extraction, recent literature, and presentation.
W1, Minor 3, Q1 & Q2 — Human Baseline Protocol, Annotator Independence, and Ethics (§3.3, Table 2, Ethics Statement)
GPT-OSS/DeepSeek(vs. top model 71.2 [Gemini-3-Flash] / 71.1 [Gemini-2.5-Flash] on the same 100 videos, paired gap −10.2 [−17.4, −3.3], 20/20 excluding zero). No evaluator took part in dataset construction, reference authoring, or guideline design.W2 & Q4 — Low MCQ Cells (
GLM-4.6V-106B,Qwen3.5-35B) and Invalid/Unparsed Rates (§4.1, Table 2, Appendix A.16) Your observation is accurate—see General Response (Part 2, §3–§5). Our initial regex matched the first standalone capital letter, returningAon 98.9% (Qwen3.5-35B-A3BT-II) and 93.3% (GLM-4.6V-106BT-III) of prose responses via"a".Qwen3.5-35B-A3Breaches 49.2/52.4 (T-II) and 76.3/80.4 (T-III;Overallrank #18 → #4), andGLM-4.6V-106Breaches 43.7/57.4 and 80.5/76.9 (#19 → #9). All cells in Table 2 now exceed uniform-random chance.N/Arate (Q4): WithN/Ascored as incorrect in the denominator, 14/20 models have 0.00%N/Aon Tasks I–III, and the max is 1.43% (Qwen3.5-9B, T-III; fullN/A& Task IV stats in General Response, Part 3, §6).Minor 1 & Q3 — Positioning Relative to Recent Works
[1–4](§2 & Table 1) Thank you for these pointers; we incorporate all four in §2 along two tracks and expand Table 1:PhyJudgescoring), showing generators frequently violate physics—motivating our use of generative videos while Video-HOCA diagnoses 20 Video-LLMs as physical reasoners across four progressive tasks (including chemistry, thermodynamics, and bio-behavior).PhysGame/GameBench) [2] (TMLR 2026) leverages synthetic gameplay glitches for tuning and MCQ testing; Video-HOCA complements it on photorealistic generative and real videos, separating Ontological vs. Causal laws, scoring 4-part explanations, and testing real-vs-generated source cues absent in homogeneous game footage.ImplausiBench[3] (2025) introduce trajectory-aware VLM tuning (complementary to Video-HOCA as an evaluation target) alongside a real-vs-generated plausibility set; Video-HOCA adds our 250-pair same-generator counterfactual control, where both normal and anomalous clips come from the same generator (General Response, Part 1, §1).Minor 2 & Artifact Release Additions
v2release (General Response, Part 3, §9) provides: (i) raw outputs, dual extractions, and judge breakdowns/sub-scores/justifications, (ii)static_dynamic_labels.jsonl(258 static / 372 dynamic), and (iii)license_manifest.jsonlacross all 1,689 unique benchmark + control clips (1,260 core + 179 probe + 250 generated normals): each clip keeps its verified upstream license (IPV-Bench/VideoPhy-2MIT,VBenchApache-2.0, ourWan2.1/HunyuanVideoclips, andPanda-70Mnon-commercial terms governing the video bundle), while annotations (CC-BY-4.0) and code (MIT) are licensed separately, with takedown support.