acceptodds
Under review as a conference paper at ICLR 2027

Video-HOCA: A Diagnostic Benchmark for Physical Anomaly Reasoning in Video-LLMs

Abstract

Video-HOCA evaluates how video-language models judge and explain visible physical anomalies. An operational Ontological-Causal taxonomy separates entity-centered from relation-centered violations, and four tasks on the same anomaly collection assess plausibility judgment, category attribution, fine-grained recognition, and mechanism explanation. The benchmark contains more than 1,439 videos and 3,470 question-answer pairs with human-verified labels and reference answers, plus a control of 250 generator-matched normal/anomalous pairs. Across 20 Instruct-mode models, nine-way category attribution remains difficult (mean Ontological/Causal macro-F1 of 37.9/46.0), and no model reaches the human reference on recognition or explanation. On the matched-generator control, the mean plausibility accuracy of 17 open-weight models is 64.4%, compared with 74.0% on the original real-versus-generated split; every model stays above chance, but the most accurate model on the original split is the least accurate on the control. For the tested model, shuffling frames barely changes plausibility accuracy yet lowers recognition and explanation scores. Annotation agreement and judge-sensitivity analyses characterize the reliability of these observations.

Primary area:
datasets and benchmarks
Keywords:
video-language models, benchmark, physical anomaly, physical commonsense, AI-generated video, video understanding, evaluation, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RB9GOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Video-HOCA, a diagnostic benchmark for how video-LLMs judge and explain everyday physical anomalies, organized by a two-level Ontological/Causal taxonomy with finer categories below. On a pool of 630 single-anomaly videos, most of them from IPV-Bench, it builds four tasks (binary plausibility against real Panda-70M clips, nine-way category attribution, four-option recognition, and open-ended explanation scored by an LLM judge), plus a control set in which normal and anomalous videos come from the same generator. Across 20 models, recognition is much stronger than attribution and no model reaches the human reference; on the generator-matched control, every open-weight model loses accuracy and the best model on the original split becomes the worst. Further analyses cover frame shuffling, Thinking, frame count, and judge and rubric robustness, some of them on only one to four models.

Critical strengths

Solid Experiments On a same-generator control (Tables 3–4), every open-weight model loses accuracy, the ranking changes substantially, and the top model on the original split becomes the worst, though all models stay above chance. This shows that real-vs-generated benchmarks partly reward source detection, which is useful for the whole community.

Reliability checks and reproducibility are strong. A second judge and a human check agree closely with the main judge, scores are insensitive to response length and rubric wording, and the judge prompt and z-score reference statistics are published (A.14–A.15). Code and data are available, although raw model outputs, the static/dynamic split file and the per-clip license manifest are not released.

Critical weaknesses
  1. Main concern is human label. The human baseline is not explained at all: the number of people, the number of items and their professional background are not given, nor is it said whether these people are the ones who wrote the reference answers.
  2. Several cells of GLM-4.6V-106B and Qwen3.5-35B are below chance, but their Task IV scores are normal, which suggests the cause is more likely an answer-extraction/format problem than an ability problem; the paper does not report the invalid/unparsed rate.

Additional (minor) issues

1 The related-work survey is not thorough: the most recent paper compared in Table 1 was released in May 2025, and there is no comparison with more recent papers. PhyWorldBench [1] (ICLR 2026 Oral) and Order from Chaos [2] (TMLR 2026), TRAVL [3] and PhyGround [4]. Many other papers can also be found. 2. The main text has only two tables. The generator-matched control (Tables 3–4), arguably the paper's main result, appears only in the appendix; it should be in the main text. 3. The paper does not describe how the human annotators were recruited, whether they were compensated, or whether informed consent was obtained (Ethics Statement).

References:

[1] Jing Gu, Xian Liu, Yu Zeng, et al. PhyWorldBench: A Comprehensive Evaluation of Physical Realism in Text-to-Video Models. In ICLR, 2026 (Oral). https://openreview.net/forum?id=rlZeILv3fm

[2] Meng Cao, Haoran Tang, Haoze Zhao, et al. Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos. Transactions on Machine Learning Research (TMLR), 2026. https://openreview.net/forum?id=Oe5TdpPv1b

[3] Saman Motamed, Minghao Chen, Luc Van Gool, and Iro Laina. TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility. arXiv:2510.07550, 2025. https://arxiv.org/abs/2510.07550

[4] Juyi Lin, Arash Akbari, Yumei He, et al. PhyGround: Benchmarking Physical Reasoning in Generative World Models. arXiv:2605.10806, 2026. https://arxiv.org/abs/2605.10806

Questions for authors
  1. Annotation agreement. What is the protocol of the human baseline in Table2 ?
  2. Three annotators re-labeled 120 stratified anomalous videos using the frozen guideline. Are these 3 annotators independent of the writers of the labels and reference answers? If these 3 people are the authors, or the group who wrote the guideline and did the data review, they received the same training, so κ would be inflated and cannot represent the agreement of external annotators.
  3. Compared with the recent work in [1–4], what are your advantages?
  4. Report the invalid/unparsed rate of Table 2.
3 comments
AuthorsOpenReview ↗

Response to Reviewer RB9G

We thank Reviewer RB9G for the encouraging assessment—highlighting our generator-matched control and reliability analyses—and for helpful suggestions on human-protocol details, option extraction, recent literature, and presentation.

W1, Minor 3, Q1 & Q2 — Human Baseline Protocol, Annotator Independence, and Ethics (§3.3, Table 2, Ethics Statement)

  1. Human baseline in Table 2 (W1, Q1; full paired CIs in ALLy W3):
    • Task III & IV (2 / 2 independently recruited evaluators holding ≥ bachelor's degree in engineering, each completing 100-video subsets marked in Table 2): On Task III, human ACC is 96.0% ([92.5%, 99.0%]) and branch macro-F1 is 95.3% ([90.1%, 98.7%]; 94.1 / 96.1) vs. top model 90.0% ACC / 90.3% F1 on the same 100 items (paired gap −6.0 [−13.0, +0.5] ACC / −4.9 [−13.0, +2.6] F1); on Task IV, human responses re-judged under our finalized rubric score 81.43 ([76.9, 85.7]) / 81.39 ([76.8, 85.6]) under GPT-OSS / DeepSeek (vs. top model 71.2 [Gemini-3-Flash] / 71.1 [Gemini-2.5-Flash] on the same 100 videos, paired gap −10.2 [−17.4, −3.3], 20/20 excluding zero). No evaluator took part in dataset construction, reference authoring, or guideline design.
  2. Independence of the 3 relabeling annotators (Q2; §3.2): Yes—the three annotators who re-labeled the 120 stratified clips (κ = 0.821 L1, 0.708 L2) are independently recruited external annotators holding ≥ bachelor's degree in engineering uninvolved in authoring references or the guideline; they annotated from the frozen guideline alone.
  3. Recruitment, compensation, and consent (Minor 3): All evaluators and relabeling annotators were recruited outside the author team, fairly compensated, and gave informed consent beforehand (§3.3 and Ethics Statement).

W2 & Q4 — Low MCQ Cells (GLM-4.6V-106B, Qwen3.5-35B) and Invalid/Unparsed Rates (§4.1, Table 2, Appendix A.16) Your observation is accurate—see General Response (Part 2, §3–§5). Our initial regex matched the first standalone capital letter, returning A on 98.9% (Qwen3.5-35B-A3B T-II) and 93.3% (GLM-4.6V-106B T-III) of prose responses via "a".

  • Under our updated extractor (360/360 human verification agreement), Qwen3.5-35B-A3B reaches 49.2/52.4 (T-II) and 76.3/80.4 (T-III; Overall rank #18 → #4), and GLM-4.6V-106B reaches 43.7/57.4 and 80.5/76.9 (#19 → #9). All cells in Table 2 now exceed uniform-random chance.
  • Table 2 N/A rate (Q4): With N/A scored as incorrect in the denominator, 14/20 models have 0.00% N/A on Tasks I–III, and the max is 1.43% (Qwen3.5-9B, T-III; full N/A & Task IV stats in General Response, Part 3, §6).

Minor 1 & Q3 — Positioning Relative to Recent Works [1–4] (§2 & Table 1) Thank you for these pointers; we incorporate all four in §2 along two tracks and expand Table 1:

  1. PhyWorldBench [1] (ICLR 2026 Oral) & PhyGround [4] (2026) evaluate video generators (T2V/TI2V physical adherence and world-model Likert/PhyJudge scoring), showing generators frequently violate physics—motivating our use of generative videos while Video-HOCA diagnoses 20 Video-LLMs as physical reasoners across four progressive tasks (including chemistry, thermodynamics, and bio-behavior).
  2. Order from Chaos (PhysGame / GameBench) [2] (TMLR 2026) leverages synthetic gameplay glitches for tuning and MCQ testing; Video-HOCA complements it on photorealistic generative and real videos, separating Ontological vs. Causal laws, scoring 4-part explanations, and testing real-vs-generated source cues absent in homogeneous game footage.
  3. TRAVL & ImplausiBench [3] (2025) introduce trajectory-aware VLM tuning (complementary to Video-HOCA as an evaluation target) alongside a real-vs-generated plausibility set; Video-HOCA adds our 250-pair same-generator counterfactual control, where both normal and anomalous clips come from the same generator (General Response, Part 1, §1).

Minor 2 & Artifact Release Additions

  • Promoting control tables (Minor 2): Agreed—Tables 4–6 (250-pair controls and item-matched decomposition) and the padding audit move into §4.2 (General Response, Part 1).
  • Artifact release (Strengths note): Our v2 release (General Response, Part 3, §9) provides: (i) raw outputs, dual extractions, and judge breakdowns/sub-scores/justifications, (ii) static_dynamic_labels.jsonl (258 static / 372 dynamic), and (iii) license_manifest.jsonl across all 1,689 unique benchmark + control clips (1,260 core + 179 probe + 250 generated normals): each clip keeps its verified upstream license (IPV-Bench/VideoPhy-2 MIT, VBench Apache-2.0, our Wan2.1/HunyuanVideo clips, and Panda-70M non-commercial terms governing the video bundle), while annotations (CC-BY-4.0) and code (MIT) are licensed separately, with takedown support.

Thank you for your comment! I confirm my acceptance 3 rating. It is a good paper.

AuthorsOpenReview ↗

Thank You for Your Support and Feedback

Thank you for your positive assessment and constructive suggestions. Your feedback on the human-baseline protocol, related work, and bringing the control results into the main text helped strengthen the paper.

Reviewer_ALLyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Video-HOCA, a benchmark for evaluating physical anomaly understanding in video-language models. It consists of four tasks: plausibility judgment, anomaly category attribution, fine-grained recognition, and physical explanation. The benchmark contains 1,439 videos and 3,470 question-answer pairs, along with a separate control set of 250 generator-matched normal/anomalous pairs. The authors evaluate 20 Instruct-mode model configurations. Across 17 open-weight models, mean plausibility accuracy is 74.0% on the original mixed-source split and 64.4% on the generator-matched control. The paper also investigates annotation agreement, temporal sensitivity, decoding configurations, and LLM-judge reliability.

Critical strengths
  1. The four complementary tasks provide a systematic framework for examining different aspects of physical anomaly understanding. Evaluating these tasks on a shared anomaly collection is useful for analyzing task-specific model behavior.
  2. The matched-generator control addresses an important limitation of the original real-normal/generated-anomalous evaluation. The authors also acknowledge that differences in sample composition prevent a strictly causal interpretation of the observed performance gap.
  3. The annotation agreement study, alternative LLM judges, rubric decomposition, and sensitivity experiments improve the transparency of the evaluation and provide useful context for interpreting the results.
Critical weaknesses
  1. Since Appendix A.14 contains the actual Task IV judge prompt, these inconsistencies raise concerns about the consistency of reference labels and evaluation criteria. The authors should clarify the rationale behind these assignments and examine whether similar inconsistencies occur in the benchmark annotations. Although the reported annotation agreement is reasonably high (Fleiss' κ = 0.821 for L1 and 0.708 for L2), aggregate agreement does not fully characterize ambiguity at individual category boundaries. A category-level confusion analysis would help establish whether these inconsistencies are isolated cases or reflect broader labeling ambiguity.
  2. Task IV allocates 25 points to scene understanding, 25 to anomaly identification, 15 to anomaly-type classification, and 35 to physical reasoning. Consequently, a model can receive 65/100 without earning any points for its physical explanation. This concern is illustrated by Example 3 in Appendix A.14 (p. 30), where a response receives 65/100 despite incorrectly attributing spontaneous rotation to gravity. Such scores may overstate physical reasoning ability when interpreted without the component-level breakdown. While Table 6 reports aggregate component scores, it does not establish how frequently models correctly identify an anomaly but fail to explain its physical mechanism. An item-level analysis separating anomaly identification from explanation correctness would better support the paper's diagnostic claims. The scoring rubric also requires clarification. Examples in Appendix A.14 suggest that the 35 reasoning points comprise 15 points for identifying the violated principle and 20 for describing the expected normal behavior, but this allocation is not explicitly stated in the rubric summary. The authors should formalize these criteria and consider a controlled analysis that varies explanation correctness while keeping the visual description fixed.
  3. Section 3.3 reports a human accuracy of 96.0% on Task III, whereas Table 2 reports branch-wise macro-F1 for the evaluated models. Since accuracy and macro-F1 are different metrics, the reported values do not support a direct quantitative human-model comparison. The human evaluation protocol also lacks essential details, including the number of participants, the number of questions answered per participant, their possible involvement in constructing reference annotations, and whether humans and models received equivalent inputs and instructions. The separate evaluation of 100 human-scored Task IV responses addresses judge reliability but does not clarify the Task III human baseline. I recommend reporting human and model performance on the same evaluation items using a consistent metric, preferably with uncertainty estimates. Otherwise, the conclusions regarding human-model performance gaps should be qualified.
  4. The matched-control dataset card [1] describes aspect-preserving resizing from 960 × 544 to 512 × 320 for certain Hunyuan normal-control videos, potentially introducing padding. It remains unclear whether such artifacts systematically differ between normal and anomalous videos or affect model predictions. The authors should verify that both classes undergo consistent spatial preprocessing and rule out label-correlated visual cues. Furthermore, the original and controlled splits differ in sample composition. An item-matched comparison using the same 250 anomalous videos with different normal counterparts would help isolate the effect of video-source differences.
  5. IPV-Bench [2] already considers hierarchical physical anomaly categories, multiple-choice recognition, open-ended explanations, and human-based evaluation. Moreover, 375 of the 630 core single-anomaly videos in Video-HOCA originate from Impossible. Although the proposed controls and diagnostic tasks are useful, the paper should more clearly demonstrate the additional insights obtained beyond existing benchmarks. For example, Qwen3.5-35B-A3B scores 21.9/1.3 on Task II and 32.7/25.3 on Task III, but 75.8/61.1 on Task IV. An error analysis separating semantic failures from response-format or parsing errors would help determine whether such discrepancies reflect genuine differences in physical reasoning ability.

Refs:

[1] Video-HOCA matched-control dataset card: Technical Alignment. (https://huggingface.co/datasets/humannobody/task1-250-pairs/blob/main/dataset/DATASET_CARD.md).

[2] Bai, Z., Ci, H., and Shou, M. Z. Impossible Videos. ICML 2025.

Questions for authors
  1. Please clarify the positive class used for F1. For Qwen3.5-122B-A10B-FP8, Tables 2 and 14 imply a normal-class F1 of approximately 87.53%, consistent with the reported 87.5%, whereas the anomaly-class F1 would be approximately 88.46%.
  2. Please specify which labels are included in the branch-wise macro-F1 calculations for Tasks II and III, and how invalid or unparseable predictions are handled.
  3. Some differences in Tables 8, 11, and 12 are relatively small. Paired confidence intervals would help assess their robustness.
  4. Appendix A.16 indicates that raw model predictions and per-response judge scores are not released. Providing these records would facilitate independent verification and error analysis.
3 comments
AuthorsOpenReview ↗

Part 1/3 — Replies to W1 & W2

We thank Reviewer ALLy for the meticulous review—especially spotting the Appendix A.14 example mismatch, suggesting the item-matched source control and controlled explanation check, and verifying exact metric definitions.

W1 — Appendix A.14 example check, benchmark-wide verification & L2 boundary patterns

  1. Appendix A.14 examples & 630-item verification (Appendix A.14): Thank you for catching this. Examples 1, 2, 4, and 5 are condensed from benchmark items (A027, A212, A304, A327) and match our taxonomy (including Example 5's Mechanical / Inertia → Causal), whereas Example 3 ("apple rotates without contact", "Ontological") was a synthetic illustration with a manual process typo. Annotators assign L2, and L1 (process) follows a fixed L2→L1 table: re-checking all 630 items confirms 0 / 630 L2→L1 mismatches in the master taxonomy used by Tasks I–III, while 6 / 630 exported Task IV reference strings in ground_truth.jsonl had unsynced Process Type text. We synchronized ground_truth.jsonl to 630 / 630 in v2 and re-aggregated all Task IV tables, sub-scores, human comparisons, and ablations from item-level outputs (20-model mean 60.18 vs. 60.14 on frozen pre-sync references, +0.04 pts, ≤ 0.14 max, 0 rank changes; General Response, Part 3, §8, with the frozen-reference control noted in Appendix A.4). Across all 20 Table 2 models (12,600 judgments), GPT-OSS-120B gave 0 partial Process scores outside {0, 15} and matched deterministic string comparison on 99.98% (12,598/12,600); holding references fixed while updating Example 3 alone yields 0 / 12,600 (0.00%) Process score changes (±0.00 mean). In Appendix A.14, we update Example 3 to "Causal" (marked synthetic), print L2/L3 tags on every example, and include the L2→L1 table.
  2. Category-level L2 confusion patterns (added to Appendix A.3 Table 8 & A.8): Building the symmetric 9×9 inter-annotator L2 matrix directly from the 120 × 3 external-annotator table behind Fleiss' κ = 0.708 (720 ordered rater pairs) alongside the model Task II matrix (630 × 20) shows the five largest row-normalized directional disagreement rates all involve Material & Rigidity:
    • Mass ↔ Material (Mass → Matl 36% annotator / 21% model; Matl → Mass 17%), Mech ↔ Material (15% / 11%, vs. 32% model Matl → Mech), and Thermo → Material (11%), followed by Mass → State (10% / 10%), reflecting overlap across rheology, force, and state/volume (refined in Appendix A.8).
    • Conversely, models' top confusions (Property → Bio 33% vs. 4%, Chem → Thermo 24% vs. 0%) and Mechanical bias (Mass/State/Bio → Mech 21%/19%/16% vs. ≤ 5%) separate model bias from boundary overlap.

W2 — 65/100 without reasoning points, sub-score separation & controlled rubric check

  1. Explicit 15 + 20 sub-scores & item-level distribution (§4.2, Table 11, Appendix A.4): We agree Task IV Total reflects composite response quality and should be read alongside component scores when evaluating physical reasoning. Under our formalized rubric (General Response, Part 3, §8), the judge outputs two reasoning sub-scores across all 12,600 Table 2 judgments—4a Violated Principle (6.96/15 mean; DeepSeek: 6.82/15) and 4b Normal Behavior (10.08/20; DeepSeek: 9.10/20)—so 20/20 on 4b does not mask 0/15 on 4a (and incompatible law attribution on the same event, as in Example 3's "Gravity" rule, is scored 0/15):
    • Sub-score distribution across Anomaly thresholds: Partitioning all 12,600 responses into anomaly-identified (Anomaly ≥ 15/25, 52.5%) and its complement (Anomaly < 15/25, 47.5%), 76.5% of identified responses score full 15/15 on 4a, while 14.9% (7.8% of all responses) score 0/15 on violated principle (3.0% of all score 0/35 total Reasoning); under stricter thresholds Anomaly ≥ 20 (47.7%) or 25 (40.8%), 4a = 0/15 is 12.7% (6.0%) or 10.6% (4.3%). The Anomaly < 15 complement averages 29.9/100 total (17.2 Scene, 0.8 Anomaly, 8.7 Process, 3.2 Reasoning). We add 4a/4b to Table 11 and foreground them in §4.2.
  2. 150-item controlled explanation check (Parts 1–3 fixed to reference text at 25 + 25 + 15 = 65 pts, varying only Part 4; added to Appendix A.4, Table 12):
    • Own reference explanation: Reasoning 35.0 / 35 (Total 100.0)
    • Fluent physics explanation swapped from another item: Reasoning 0.2 / 35 (Total 65.2; < reference on 150/150 items)
    • Generic boilerplate ("violates physical common sense..."): Reasoning 10.0 / 35 (Total 75.0)
    • Explanation omitted: Reasoning 0.0 / 35 (Total 65.0) These 65.0–75.0 totals illustrate why §4.2 pairs Total with Reasoning (4a+4b) sub-scores; omitting the 15-pt Process label also preserves 20-model rankings (Spearman ρ = 0.988, max shift 2).
AuthorsOpenReview ↗

Part 2/3 — Replies to W3, W4 & W5

W3 — Human baseline: protocol, aligned metrics & paired same-item CIs (§3.3, §4.2, Table 2) Thank you—in submitted §3.3 and Table 2, 96.0% was two-evaluator mean accuracy rather than branch macro-F1; we now align metrics, mark Table 2's Human row as 100-item stratified subsets, and report 10,000-resample video-level paired bootstrap 95% CIs (keeping both human answers per item and recomputing macro-F1 per resample). All evaluators were independently recruited outside the author team (≥ bachelor's degree in engineering, uninvolved in data construction, reference authoring, or guideline design, fairly compensated with informed consent):

  • Task III (2 evaluators each completing all 100 items, 44 Onto / 56 Caus, under identical videos, options, and instructions; 95.0% & 97.0% ACC, 96.0% agreement): Human accuracy is 96.0% ([92.5%, 99.0%]) and count-weighted branch macro-F1 is 95.3% ([90.1%, 98.7%]; 94.1 / 96.1 Onto/Caus). On the same 100 items, the 20-model mean is 77.6% ACC / 76.9% macro-F1 (79.8 / 74.6), and top model Qwen3.5-27B scores 90.0% ACC ([84.0%, 95.0%]) / 90.3% macro-F1 ([83.2%, 95.8%]; 91.8 / 89.2)—paired model − human gap −6.0 ([−13.0, +0.5]) ACC / −4.9 ([−13.0, +2.6]) macro-F1 (paired CIs exclude zero for 19/20 models on ACC and 18/20 on macro-F1). We qualify §4.2 and the Abstract accordingly.
  • Task IV (2 evaluators writing four-part responses on a 100-video subset, 39 Onto / 61 Caus, distinct from Appendix A.4 / Table 13's audit of model outputs): Re-judged under our finalized rubric and synchronized references (General Response, Part 3, §8), humans score 81.43 ([76.9, 85.7]; 85.6 / 78.8) under GPT-OSS-120B and 81.39 ([76.8, 85.6]; 85.0 / 79.1) under DeepSeek-V4-Flash. On the same 100 videos, the 20-model mean is 60.7 / 58.9, and top models score 71.2 (Gemini-3-Flash, GPT-OSS; paired gap −10.2 [−17.4, −3.3]) and 71.1 (Gemini-2.5-Flash, DeepSeek; paired gap −10.2 [−17.3, −3.2]; 20/20 models exclude zero under both judges)—confirming a significant human lead on composite explanation quality, with higher point estimates on Reasoning as well (26.6 / 25.1 vs. ≤ 22.9 / 22.3 out of 35).

W4 — Spatial preprocessing / padding audit & item-matched comparison (§4.2 & Appendix A.5)

  1. 250-pair item-matched comparison & padding audit (General Response, Part 1, §1–§2; moved to §4.2): Thank you for flagging the dataset card note. Sharing the identical 250 anomalous-clip predictions across both rows (recall 54.1%) isolates the entire −8.3 shift (72.7% → 64.4%) to normal-half false alarms (91.4% → 74.8%). Across all 500 control clips, container metadata matches 250/250 (AUC = 0.500), while pixel inspection found letterboxing/pillarboxing on 4.0% (10/250) of normal vs. 0.0% (0/250) of anomalous clips (5/8 512×320 HunyuanVideo pairs plus 5 Wan2.1 pairs). Replacing those 10 normal clips with native-source center-cropped versions across all 17 open-weight models (holding the 490 unmodified clips fixed) gives ΔACC = +0.02% (+2/8,500; ΔF1 = +0.02%; 12/170 flips on the 10 clips: 7 turned correct, 5 wrong), showing negligible net impact.
  2. Original 1,260-clip container check (added to Appendix A.5): On the mixed-source 1,260-clip split (where all 630 real normals are 720p vs. 380/630 anomalous clips), uniform border-cropped 720p re-encoding and random cross-class container-property permutation produce no statistically significant Task I change across all four paired McNemar cells: +0.16% (p = 0.85) and −0.87% (p = 0.14) on Qwen3-VL-32B, and −0.56% (p = 0.36) and −0.08% (p = 1.00) on Qwen3.5-27B (at aligned 1,024-token budget).

W5 — Positioning relative to IPV-Bench & separating semantic vs. extraction effects (§2 & §4.2)

  • Extraction vs. model capability: As shown in General Response (Part 2–3), Qwen3.5-35B-A3B reaches 49.2/52.4 (T-II) and 76.3/80.4 (T-III; N/A ≤ 0.79%) under our updated extractor, consistent with Task IV.
  • Complementary scope to IPV-Bench (§2): Impossible Videos (IPV-Bench) established counterfactual benchmarking across physical, biological, social, and geographical domains. Video-HOCA complements it on macro-physical diagnosis (annotating all 375 reused clips from scratch) via: (i) 3-tier entity-vs-interaction granularity (9 L2, 18 L3; IPV-Bench's "physical laws" spans 8 of our 9 L2 groups); (ii) 4 progressive tasks on shared clips—contrasting 4-option event recognition (78.8/72.2 20-model F1, chance ~25%), 9-way law attribution (40.2/51.5 F1, chance 15.8/11.8%), and 4-part explanation (where 14.9% of anomaly-identified responses score 0/15 on violated principle), alongside static/dynamic stratification (Qwen3.5-122B, YdBB Q1); and (iii) 250-pair same-generator controls (General Response, Part 1).
AuthorsOpenReview ↗

Part 3/3 — Replies to Q1–Q4

Q1–Q4 — Specific Questions

  • Q1 (Task I F1 positive class; §3.3, Table 2 & Table 4 captions): Confirmed—Table 2 used the normal class (gt = Yes) as positive (87.53% for Qwen3.5-122B; anomaly-positive F1 is 88.46%, ACC 88.02%), whereas control Table 4 reported anomaly-positive (gt = No) F1 (68.2% for Qwen3.5-122B; normal-positive F1 is 43.0%). We state both conventions explicitly in §3.3 and the Table 2 / Table 4 captions and report both F1 variants (Appendix Table 9).
  • Q2 (Macro-F1 labels & N/A across tables; §3.3 & Appendix A.16): Branch macro-F1 weights equally all ground-truth labels in that branch (Task II Onto: State, Property, Bio-Behavior; Caus: Mechanical, Material, Mass & Vol., Thermodynamic, Chemical, Optical; Task III: A–D). Out-of-branch and N/A predictions count as false negatives for the true class (never dropped). Beyond Table 2 (≤ 1.43%, General Response, Part 3, §6), MCQ N/A is 0.00% on Tables 4–5, 23 & Table 17 Instruct/Instruct-Align, ≤ 0.95% on Table 22 (64f), and up to 13.97% under Thinking (Qwen3.5-122B T-II; other Thinking runs ≤ 3.17%).
  • Q3 (Paired 95% CIs for Tables 17, 22, 23 & updated prose in §4.3, Appendix A.6, A.11, A.12; item-paired bootstrap recomputing ACC for T-I, count-weighted branch macro-F1 per resample for T-II–III, and judge score for T-IV; † = 95% CI excludes 0):
    • Table 23 (Qwen3.5-122B): 1f − 16f excludes zero on T-I (−7.38 [−9.44, −5.24]†), T-III macro-F1 (−8.34 [−11.65, −5.26]†), and T-IV (−6.32 [−9.73, −2.96]†), while pooled T-II macro-F1 spans zero (−2.69 [−6.12, +0.82], with dynamic Onto at −8.3 [−14.2, −2.2]†; per-item T-II ACC −6.03 [−9.52, −2.54]†); Shuffled − 16f excludes zero on T-III macro-F1 (−3.73 [−6.01, −1.60]†) and T-IV (−3.01 [−5.36, −0.70]†), and spans zero on T-I (−0.08) and T-II macro-F1 (−2.73 [−5.78, +0.26]).
    • Table 17 (Thinking − Instruct; Instruct-Align = Instruct checkpoint at Thinking's sampling budget): Thinking improves GLM-4.6V-106B T-IV (+5.52 [+3.59, +7.47]†, +3.59† over Instruct-Align) and Qwen3.5-122B T-II macro-F1 (+10.36 [+6.42, +14.61]†, +13.88† over Instruct-Align), T-III macro-F1 (+3.23 [+0.33, +6.07]†), and T-IV (+3.87 [+1.45, +6.30]†, +5.32† over Instruct-Align), lowers T-I on Qwen3.5-122B (−4.13†) and Qwen3-VL-32B (−4.76†), and spans zero on GLM & Qwen3-VL-32B T-II/III macro-F1.
    • Table 22 (64f − 16f): Under count-weighted branch macro-F1 (and ACC), only Gemini-2.5-Flash T-I excludes zero (−5.40 [−7.47, −3.42]†); 15 of 16 cells span zero. We refine §4.3 (Finding 5) and Appendix A.11 accordingly (General Response, Part 3, §7) and include CIs in all three tables.
  • Q4 (Raw outputs & judge scores): Provided in predictions/ of our v2 release (General Response, Part 3, §9).
Reviewer_YdBBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Video-HOCA, a diagnostic benchmark designed to evaluate Video Large Language Models (Video-LLMs) on their judgment and reasoning regarding visible macroscopic physical anomalies. The authors construct a three-tier "Ontological–Causal" taxonomy (comprising 2 L1 branches, 9 L2 groups, and 18 L3 fine-grained labels) and design four progressive tasks on the same set of anomalous videos (630 videos in the core single-anomaly set; 1,439 videos and 3,470 QA pairs in total): Plausibility Judgment (Task I), Category Attribution (Task II), Fine-grained Identification (Task III), and Open-ended Mechanism Explanation (Task IV). Additionally, the paper constructs a counterpart set of 250 normal/anomalous video pairs from matched generators to eliminate confounding factors arising from discrepancies between real and synthetic video sources. The authors conduct extensive evaluations across 20 instruction-tuned models and perform experimental analyses on annotation agreement, multi-judge sensitivity, frame-order ablations, and decoding configurations (Thinking vs. Instruct).

Critical strengths
  1. Clear and well-structured hierarchical diagnostic task design. Rather than stopping at binary classification or generic QA, the paper leverages an "Ontological–Causal" taxonomy to decompose evaluation on the same set of anomalous videos into four progressive tasks: plausibility judgment, category attribution, fine-grained identification, and mechanism explanation. This design effectively pinpoints whether a model fails at perceiving the anomaly or understanding the underlying physical laws, offering a clear diagnostic logic.
  2. Strong open-science practices and reproducibility. The authors release the complete dataset, the generator-matched counterpart set, evaluation code, prompt templates for all tasks, and scoring rubrics for Task IV, alongside detailed documentation of annotation guidelines and model decoding configurations. This greatly facilitates reproducibility and follow-up research.
Critical weaknesses
  1. Small core dataset scale and heavy reliance on splicing existing benchmarks. Only 630 single-anomaly videos are actually used for Tasks II–IV, which averages to merely ~35 videos per category when distributed across the 18 L3 labels. At this small sample scale, fine-grained evaluation is subject to high variance, yet the paper provides no category-level confidence intervals. Furthermore, up to 473 of these 630 videos are directly reused from Impossible Videos, VideoPhy-2, and VBench, with only 157 videos newly generated by the authors. Because the work focuses primarily on relabeling and reorganizing existing data, its substantive data-level contribution as a new benchmark is limited.
  2. Over-reliance on text-only models for distractor generation and evaluation, detaching from visual grounding and introducing textual shortcuts. The paper relies heavily on a blind, text-only model (GPT-OSS-120B) without visual input for both core task construction and evaluation. In Task III, distractors are fabricated solely from brief text labels, which easily produces absurd options that conflict with the visual background or exposes linguistic style biases—allowing models to eliminate incorrect options using shallow object recognition or textual priors. In Task IV, the judge model scores responses based solely on text comparison without seeing the raw video frames. Moreover, the rubric includes a generous 40-point base score (25 points for scene description + 15 points for a binary label) and instructs the judge to remain "lenient" toward generic descriptions, enabling models to game high scores using single-frame scene recognition plus boilerplate physics commonsense. Both the omission of a blind text-only baseline and the anomalous phenomenon where single-frame inputs maintain high scores on Task III (77.9/70.7) and Task IV (58.0/59.0) confirm that this text-only pipeline introduces severe evaluation shortcuts.
Questions for authors
  1. (Clarification Request) The paper claims that 59% of the videos and roughly two-thirds of the Causal samples involve dynamic anomalies requiring cross-frame evidence. However, in Table 12, when Qwen3.5-122B is fed only a single frame (1f), its scores on the Causal branch for Task II (41.0 → 40.2) and Task IV (61.4 → 59.0) show almost no degradation. Why can a static single frame maintain nearly identical causal attribution and physical explanation performance? Does this indicate that the purported dynamic samples suffer from severe single-frame visual shortcuts or scene prior leakage?
  2. (Clarification Request) In Table 2, Qwen3.5-35B-A3B and GLM-4.6V-106B collapse on Task II (Causal accuracy of only 1.3% and 21.9%, respectively) and Task III (25.3% and 18.0%), yet achieve high scores on the most challenging open-ended explanation Task IV (75.8/61.1 and 71.0/58.9). How can a model that completely fails to identify anomalies in multiple-choice questions achieve high scores in explaining physical mechanisms? Could this metric distortion be caused by option-parsing failures?
  3. (Clarification Request & Suggestion) In practice, a core pain point of anomaly detection is false alarms on normal videos. Tasks II–IV only evaluate anomalous videos, which embeds a strong prior that an anomaly must exist. If normal videos were fed into Tasks II–IV, would the models forcibly fabricate physical violations (anomaly hallucination)? Have the authors tested or considered this aspect?
5 comments
AuthorsOpenReview ↗

Part 1/3 — Reply to W1 (Reporting Granularity, Uncertainty & Data Contributions)

We thank Reviewer YdBB for the rigorous evaluation and constructive questions, which motivated new blind text-only baselines, a vision-language distractor reconstruction study, stratified temporal controls, and normal-video abstention experiments.

W1 — Dataset scale, uncertainty across reporting levels, and video + annotation contributions

  1. Scope of claims & uncertainty at the levels actually reported (§3.1, §3.3, §5, Table 2, Appendix A.3): We agree that ~35 videos per L3 tag (31–37 per tag) are insufficient for category-specific model rankings—and the paper makes no per-L3 model comparison. As stated in §3.1, all main comparisons are reported at the L1 Ontological/Causal branch level (n = 254 / 376) and over the 630-video core set; the 9 L2 groups serve for dataset balancing, annotation, and as the 9-option label space of Task II; and the 18 L3 tags organize balanced sub-phenomenon coverage and guide option/reference authoring. "Fine-grained" in Task III refers to the event-level specificity of the four candidate descriptions per video (scored per L1 branch), not to an 18-way per-L3 leaderboard; we clarify this explicitly in §3.3. (L3 enters statistical analysis only once, as a control covariate in Appendix A.5 Table 15's generator-composition adjustment pooled over 20 models × 630 items, where no per-L3 score is reported or compared.)
    • Uncertainty where conclusions are drawn: In Appendix Table 9 (with half-width ranges in the Table 2 caption) and §4.2, we add item-level bootstrap 95% confidence intervals (recomputing branch macro-F1 per resample) for all metrics alongside paired bootstrap intervals for within-task comparisons emphasized in the text (across the 20 Table 2 models, L1 branch macro-F1 half-widths are ±4.8–7.1 pts on Task II and ±3.5–6.2 pts on Task III).
    • Descriptive category profiles & scope note: Per-L2 and per-L3 20-model mean within-category recognition accuracy (with video-level bootstrap 95% intervals preserving cross-model correlation within each resampled clip, explicitly labeled as within-category accuracy rather than macro-F1) is placed in Appendix A.3 (Table 10) strictly as descriptive profiles and marked as not supporting model rankings; we also note in §5 that branch-level aggregation does not remove the finite sample size within individual L3 phenomena.
  2. Core diagnostic contribution & newly constructed video/annotation assets (§3.2): We agree that Video-HOCA's primary contribution lies in our three-tier Ontological–Causal taxonomy, four-task progressive diagnostic design on shared clips, source-deconfounded controls, and dense from-scratch annotations rather than raw clip volume alone. Even on the 473 reused core clips (§3.2: 375 Impossible Videos, 57 VideoPhy-2, 41 VBench), every clip was annotated from scratch by watching the video because upstream tags do not map to ours: Impossible Videos' "physical laws" (n = 239) spans 8 of our 9 L2 groups, its "spatial/temporal" flag aligns with our perceptual static/dynamic label on 77.3% (290/375) of clips, and our 18 L3 tags, 4-part causal explanations, and 4-task QA pairs do not exist upstream. Alongside standard reuse of validated anomalous clips, our newly generated and curated video additions are substantial: (i) 157 newly generated Wan2.1 anomalous clips balancing underrepresented L3 laws, (ii) 250 newly generated normal counterfactuals paired with 250 core anomalous clips to form our 250-pair same-generator control (absent in prior VLM benchmarks in Table 1), (iii) 179 additional clips and dense interval/checklist annotations for our temporal-grounding (200 QA) and multi-anomaly (120 clips) probes, and (iv) 630 curated, semantically matched Panda-70M real controls.
AuthorsOpenReview ↗

Part 2/3 — Reply to W2 (Blind Baseline, Task IV Audit & Visually Grounded Distractor Rebuild)

W2 — Blind text-only baseline, Task IV rubric audit, and visually grounded Task III distractors

  1. Four-task blind text-only baseline across all 17 open-weight models (identical prompts, greedy decoding, extractor, and formalized Task IV judge, frames removed; added to Table 2 & §4.2):
Input Condition (17-model mean)Task I ACCTask II (Onto / Caus)Task III (Onto / Caus)Task IV (/100)
16 frames (with video)74.040.7 / 51.679.4 / 72.658.8
Blind text-only (no video)50.04.1 / 6.728.9 / 46.26.9
Uniform random50.015.8 / 11.824.5 / 25.1—

(Task II blind sits below uniform random because every blind model receives identical text across all 630 items and selects a single fixed category letter, yielding 0 F1 on 8 of 9 classes.) 2. Testing the "40-pt base" & seen-scene boilerplate on Task IV (Appendix A.4): Six blind models abstain (0–0.2 pts); across the 11 that answer without video, mean scores under our formalized rubric (General Response, Part 3, §8) are Scene 0.87/25, Anomaly 0.39/25, Reasoning 1.13/35, and Process 8.19/15 (10.6/100 total)—confirming Scene points require visual input. For seen-scene + boilerplate reasoning, fixing Parts 1–3 to gold scene/anomaly/process (65 pts) shows generic physics boilerplate earns 10.0/35 on Reasoning and cross-item physics earns 0.2/35 (ALLy W2); we pair Task IV totals with component/sub-score breakdowns in §4.2 and Table 11. Dropping the Process label leaves rankings highly consistent (Spearman ρ = 0.988, max shift 2), as does removing both brevity clauses on 500 items (Pearson r = 0.940, Spearman ρ = 0.953). 3. 2×2 study of our visually constrained & audited Task III reconstruction (DeepSeek-V4.1-Flash): We note in §5 that GPT-OSS-120B generated the original distractors from text metadata. Following your suggestion, we built a 630-item visually constrained and audited reconstruction (keeping gold options verbatim): feeding 16 frames into DeepSeek-V4.1-Flash, enforcing same-L1 / distinct-L3 distractors anchored to visible objects, length parity (max/min ≤ 1.6; balanced A–D 158/158/157/157), two-pass VLM non-occurrence screening, and a 100-item random human audit checking out-of-scene objects (0/100 violations), distractor non-occurrence, and gold validity. Across all 17 open-weight models in a same-machine evaluation (17-model mean within 0.2 pts of Table 2, 75.5 vs. 75.3; ρ = 0.988 vs. Table 2):

Task III Set (17-model same-machine run, count-weighted F1; random ≈ 24.8)Blind (No Video)With Video (16f)Video Gain
Original set (GPT-OSS-120B, text-only)39.2 (28.7 / 46.3)75.5 (79.8 / 72.6)+36.3
Visually constrained & audited rebuild (DeepSeek-V4.1-Flash, 16f)32.4 (29.2 / 34.5)69.0 (74.0 / 65.5)+36.6
  • Video gain preserved & option-design sensitivity quantified: The rebuilt set cuts the above-chance blind margin roughly in half (+14.4 → +7.6) while preserving the full video gain (+36.3 → +36.6; +28.0 to +44.8 per model). With-video scores move from 75.5 to 69.0 with rank correlation ρ = 0.782 (Pearson r = 0.855): Qwen3.5-27B remains #1 (87.7 → 79.3), whereas models with higher original blind scores move down (e.g., Qwen3.5-9B #3 → #10, −12.21 pts; Qwen3.5-122B #2 → #8, −12.18 pts) and lower-blind-score models move up (e.g., Qwen3-VL-8B #5 → #2; Qwen3-VL-30B-A3B #8 → #4)—consistent with your observation that text priors affect part of the original ordering and showing sensitivity to distractor design.
  • Blind scores across model families: On the rebuilt set, blind F1 is near chance for InternVL-2.5 / Qwen2.5-VL (25.0), InternVL-3.5 (26.5), and Qwen3-VL (31.1), vs. 37.9 (GLM-4.6V) and 42.0 (Qwen3.5; items answered by ≥ 9/17 blind models drop from 242/630 to 177/630), possibly reflecting stronger LLM priors over common generator patterns.
  • Revision & release (§4.2, Tables 2–3, Appendix A.9 Table 19): On the original set, removing those 242 blind-screened items (n = 388 blind-screened subset) gives ρ = 0.975 vs. full original. To preserve 630-video cross-task continuity while surfacing option sensitivity directly, we retain the original column in Table 2 alongside a blind-baseline row, add this 2×2 table (Table 3) in §4.2 (with all 17-model rebuilt scores/ranks and the 388-item blind-screened check in Appendix A.9, Table 19), qualify fine-grained Task III rank discussion in §4.2, and release the rebuilt 630-item set and blind flags.
AuthorsOpenReview ↗

Part 3/3 — Replies to Q1, Q2 & Q3

Q1 — Why does 1 frame retain Task II Causal (41.0 → 40.2) and Task IV Causal (61.4 → 59.0 submitted; 59.1 → 57.6 under our formalized rubric, General Response, Part 3, §8)? Stratifying this single-model (Qwen3.5-122B) ablation by static_dynamic_labels.jsonl (258 static, 372 dynamic; 10,000 item-paired bootstrap resamples, † = 95% CI excludes 0) shows a clear static-vs-dynamic divergence: single-frame cues suffice on static clips, whereas the dynamic majority (372/630) drops substantially across visual recognition and explanation:

  1. Overall 1f − 16f shifts (Table 23): Per-item accuracy drops on T-I (−7.38 [−9.44, −5.24]†), T-II (−6.03 [−9.52, −2.54]†), and T-III (−7.94 [−11.11, −4.76]†; count-weighted macro-F1 −8.34 [−11.65, −5.26]† on T-III, −2.69 [−6.12, +0.82] on pooled T-II), and judge score drops −6.32 [−9.73, −2.96]† on T-IV (Onto −13.4).
  2. Stratified 1f − 16f Δ [95% CI] (Qwen3.5-122B, single-model; per-item accuracy [0/100] for T-I–III, 0–100 judge score for T-IV; added to Appendix A.12, Table 24):
Subset (1f − 16f)T-I ACC (Onto / Caus)T-II ACC (Onto / Caus)T-III ACC (Onto / Caus)T-IV Score (Onto / Caus)
Static (130 O / 128 C)−5.4† / −16.4†+0.0 / +4.7+4.6 / −4.7+15.5 [+9.3,+21.5]† / +12.4 [+6.7,+18.1]†
Dynamic (124 O / 248 C)−24.2† / −24.2†−22.6† / −6.5 [−12.1,−0.8]†−19.4† / −10.5†−43.6 [−50.5,−36.7]† / −8.8 [−13.7,−4.0]†
  • Opposite static vs. dynamic shifts explain the pooled Causal numbers: Static Task IV rises +14.0 [+9.8, +18.4]† (+12.4† Causal, possibly because one clear frame reduces multi-frame distraction), whereas dynamic clips drop −20.4 [−24.6, −16.1]† (−8.8† Causal; dynamic Onto Anomaly 19.0 → 5.2, Reasoning 25.0 → 8.9), averaging out to −1.6 on pooled Causal. On dynamic Task II, branch macro-F1 drops on Ontological (−8.3 [−14.2, −2.2]†), while on Causal per-item accuracy drops −6.5 [−12.1, −0.8]† vs. −0.8 [−7.8, +6.0] in branch macro-F1—reflecting a strong A: State prediction bias on dynamic Causal (31.9 F1 at 16f; 55.6% [138/248] A: State predictions and 44.4% answer flips at 1f).
  1. Shuffled (16f shuffled − 16f) & blind controls: Shuffling 16 frames leaves static clips unchanged (T-IV +0.0 [−3.4, +3.5]) while lowering dynamic clips on T-III (−4.8 [−7.8, −2.2]†) and T-IV (−5.1 [−8.2, −2.1]†; overall −3.65†, −3.01†); removing frames yields 7.7 on T-IV. In §4.3 and Appendix A.12 (Tables 23–24), we keep overall rows and add both accuracy and macro-F1 stratified CIs (noting single-model scope).

Q2 — Task II/III low scores vs. strong Task IV: option-extraction issue? Yes. As detailed in General Response (Part 2, §3–§5; Part 3, §6), our initial regex matched "a" in prose as A on 98.9% (Qwen3.5-35B-A3B T-II) and 93.3% (GLM-4.6V-106B T-III) of responses. Under our updated extractor (N/A ≤ 1.43%), Qwen3.5-35B-A3B scores 49.2/52.4 (T-II) and 76.3/80.4 (T-III; Overall rank #18 → #4), and GLM-4.6V-106B scores 43.7/57.4 and 80.5/76.9 (#19 → #9), aligning with Task IV.

Q3 — Anomaly hallucination on normal videos in Tasks II–IV Normal-video false alarms are evaluated directly in binary Task I (General Response, Part 1, §1), whereas Tasks II–IV are conditional diagnoses given an anomalous clip (stated explicitly in §3.3). To examine Tasks II–IV on normal videos without forcing a choice, we tested 6 models on the 250 generated normal counterfactuals (paired with their anomalous clips' options) after adding "J. / E. No physical anomaly is present" in T-II/III and a neutral T-IV prompt allowing "No anomaly" (added to Appendix A.10):

Model (250 generated normal counterfactuals)Task II (J: No anomaly)Task III (E: No anomaly)Task IV (No anomaly)
Qwen3-VL-8B / Qwen3-VL-32B78.8% / 78.0%85.6% / 83.6%99.6% / 86.8%
Qwen3.5-27B / GLM-4.6V-9B-Flash90.4% / 79.6%58.8% / 96.4%88.0% / 59.2%
InternVL-3.5-38B / Qwen3-VL-2B81.6% / 16.8%78.0% / 50.8%34.8% / 81.6%
6-model mean abstention70.9%75.5%75.0%

Non-abstentions are almost entirely explicit anomaly selections/claims (29.1% / 24.4% / 24.6%, vs. 0.0% / 0.1% / 0.4% N/A/ambiguous). Models thus abstain on 70.9%–75.5% of generated normals—closely tracking the same 6 models' 73.4% accuracy (26.6% false alarms) on those identical 250 generated normals in Task I (vs. 89.7% on real normals)—with notable T-IV spread (Qwen3-VL-8B 0.4% vs. InternVL-3.5-38B 65.2%). We release these escape-option variants in v2.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Official Comment to All Reviewers (General Response) Part 1/3 — Generator-Matched Control: Item-Matched Decomposition & Preprocessing Audit

We thank ALLy (Strengths, W4, W5) and RB9G (Strengths, Minor 1–2, Q3) for highlighting our 250-pair (500-video) same-generator counterfactual control as a central result, as well as YdBB (W1, Q3) on counterfactual data contributions and normal-video false alarms. Table and appendix numbers below follow the revised PDF (changes in blue; submitted Tables 3, 4, 6, 7, 8, 10b, 11, and 12 are now Tables 5, 4, 11, 15, 17, 21b, 22, and 23).

1. Item-matched normal/anomalous decomposition (ALLy W4, RB9G Minor 2, YdBB Q3). Because comparing the full mixed-source split (630 + 630, 74.0% ACC) against the 250-pair control (64.4% ACC) also changes the anomalous subset, we hold the exact same 250 anomalous clips fixed to control anomalous-sample composition and compare performance across all 17 open-weight models when paired with either their matched real Panda-70M normals or their same-generator (Wan2.1 / HunyuanVideo) normal counterfactuals (sharing the identical Table 4 anomalous-clip predictions across both rows, recall 54.1%; Task I binary Yes/No outputs are unaffected by the MCQ extraction update in Part 2):

Paired normal source (same 250 anomalous clips, 17 open-weight models)Overall ACCNormal-half ACC
Real (Panda-70M)72.791.4
Same-generator counterfactual64.474.8
Δ (Same-generator − Real)−8.3−16.6
  • Low anomaly recall (54.1%, near the 50% uniform-guessing reference) reflects a plausibility prior masked by real normals: Mean recall on these 250 anomalous clips is only 54.1% (28.0%–87.6% across models under a strong "Yes" plausibility prior), showing that the 72.7% (74.0% full-split) headline largely reflects easy acceptance of real normal clips (91.4%)—directly motivating our conditional diagnostic Tasks II–IV on anomalous videos.
  • The entire −8.3 shift comes from generated normals (−16.6 on the normal half): With the 250 anomalous predictions held fixed across both rows, normal-half accuracy drops from 91.4% to 74.8%, showing that real-vs-generated splits substantially reward source cues on the normal class. The top model on the original split (Qwen3.5-122B, #1 at 88.0% full / 86.4% item-matched) ranks #17 here (59.2%), as its normal-half accuracy shifts from 85.2% to 30.8% (Δ = −54.4; anomalous recall 87.6%).
  • Physical signal persists after source deconfounding: On the 250 pairs, 10,000-resample matched-pair bootstrap 95% CIs exclude 50% for all 17 / 17 models (59.2%–69.6%, lowest lower bound 56.2%), with above-chance point accuracy on both generators (Wan2.1 60.5%–69.8%, HunyuanVideo 51.8%–75.3%) and on dynamic clips (62.4% mean).

2. Control-set spatial preprocessing & padding audit (ALLy W4). Pair-by-pair container checks across all 500 control clips confirm exact metadata parity (0/250 mismatches in width, height, codec, pixel format, or fps; metadata label AUC = 0.500). Pixel-level inspection cross-checked against native generator resolutions finds letterboxing/pillarboxing after aspect-preserving resizing on 4.0% (10/250) of normal and 0.0% (0/250) of anomalous clips (5 of the 8 512×320 HunyuanVideo pairs resized from 960×544, plus 5 Wan2.1 pairs). Rebuilding those 10 normal clips from native sources via uniform aspect-preserving center-cropping (0 border pixels across both classes, all other encoding parameters identical) and replacing them across all 17 open-weight models (holding the 490 unmodified clips fixed) shifts 500-clip Task I by ΔACC = +0.02% (+2 / 8,500 predictions; ΔF1 = +0.02%; on those 10 clips, 12 / 170 predictions flip—7 turned correct, 5 turned wrong), showing negligible net impact on overall scores.

In the revised PDF (changes marked in blue), Tables 4–6 (promoted control tables and this item-matched decomposition) and the padding audit appear in §4.2 (shifting secondary sensitivity prose from §4.4 to Appendix A.4/A.5), and the Abstract reports the item-matched comparison.

AuthorsOpenReview ↗

Part 2/3 — Answer Extraction Alignment & Human Verification

We also thank Reviewers YdBB (Q2), ALLy (W5, Q2), and RB9G (W2, Q4) for independently noting a second shared pattern: verbose models such as Qwen3.5-35B-A3B and GLM-4.6V-106B-FP8 showed near- or below-chance scores on multiple-choice Tasks II–III alongside strong open-ended explanation scores on Task IV, raising the question of option extraction and invalid/unparsed rates. All three reviewers' observation is accurate: this discrepancy arose from our original option-extraction rule rather than the models' underlying outputs. We summarize our unified extraction update here and cross-reference it in individual replies.

3. Cause of the extraction discrepancy. Our initial Task I–III script upper-cased the raw response and took the first standalone capital letter in the valid option range. When models wrote full prose sentences before stating a choice (e.g., "The video shows a ...", "To determine a broad category ...", or "I think ..."), this rule matched the English article "a" or pronoun "I" as option A or I. On stored raw outputs, it assigned A to 98.9% of Qwen3.5-35B-A3B's Task II responses (87.3% on Task III) and 93.3% of GLM-4.6V-106B-FP8's Task III responses (76.8% on Task II), against ground-truth A priors of 17.3% and 27.0%. Because most English sentences contain "a" or "I", unparseable outputs also mapped to an option letter, leaving the apparent invalid rate at zero.

4. Updated extraction pipeline & adjudication rule (added to §4.1 & Appendix A.16). Without re-running model inference, we re-extracted all stored responses across every evaluation setting using a blind LLM extractor (GPT-OSS-120B, temperature = 0) as primary extractor, cross-checked against strict deterministic templates:

  • Primary blind LLM extractor (GPT-OSS-120B): Given the question and the model's post-reasoning response without ground truth, asked solely which option the response commits to (or N/A); correctness is decided deterministically in code.
  • Cross-check & deterministic adjudication: Strict templates match only explicit answer forms (boxed letters, standalone option lines, or "the answer is X"), returning N/A otherwise. Across all unique single-choice responses:
    1. Outright agreement (89.4% of all unique responses; 99.6% where strict templates return a valid option): Adopted directly.
    2. Strict template = N/A, blind LLM = valid option (10.2%): Verbose responses concluding with a clear choice outside fixed regex forms (e.g., stating an option description or trailing letter after elimination); the blind LLM extraction is adopted, while unclosed/truncated outputs lacking a final commitment statement are deterministically set to N/A.
    3. Direct disagreements (0.38%: 0.23% strict = valid vs. LLM = N/A; 0.15% different letters): Closed responses take the blind LLM's reading of the post-reasoning answer; truncated unclosed traces require three consistent explicit commitment statements at the end and otherwise map to N/A.
  • Ground-truth visibility check: Providing ground truth to GPT-OSS-120B agrees with our blind run on 99.5% of unique items and shifts no metric by more than 0.66 pts, showing no noticeable reference-answer pull.

5. 360-item stratified human verification (Appendix A.16). We manually audited 360 final extractions across three independent strata—Stratum A: 100 uniform-random Task I–III rows; Stratum B: 200 oversampled verbose or N/A Task I–III rows; Stratum C: 60 auxiliary multi-anomaly rows. Reporting per-stratum Wilson 95% CIs under our post-reasoning scoring protocol, human agreement is 100 / 100 on random Task I–III ([96.3%, 100.0%]), 200 / 200 on verbose/N/A Task I–III ([98.1%, 100.0%]), and 60 / 60 on Stratum C ([94.0%, 100.0%]). On Tasks I–III, a sensitivity check crediting conclusions inside unclosed <think> blocks changes only Thinking-mode runs by ≤ 0.6 pts (0.0 on Table 2) and alters no ranking.

AuthorsOpenReview ↗

Part 3/3 — Invalid (`N/A`) Rates, Updated Results, Task IV Re-Scoring & Release

6. Invalid (N/A) rates, old-vs-new scope & Task IV validity (ALLy Q2, RB9G Q4; added to §4.1 & Appendix A.16). On Tasks I–III, N/A (refusal, multiple choices, pre-conclusion cutoff, or self-contradiction) is scored as incorrect and kept in the denominator. In Table 2, 14/20 models have 0.00% N/A across Tasks I–III, and all 20 are ≤ 1.43%:

Non-zero N/A models (Table 2)Task I (%)Task II (%)Task III (%)
Qwen3.5-9B / GPT-4o0.00 / 0.630.79 / 1.111.43 / 1.11
Qwen3.5-35B-A3B / Gemini-2.5-Flash0.00 / 0.000.16 / 0.000.79 / 0.32
Qwen3.5-27B / Qwen3-VL-8B0.00 / 0.000.16 / 0.000.00 / 0.16
  • Old-vs-new scope: Table 2 extractions differ between the initial and updated pipelines on 9.19% of T-II/III items (4.63% across T-I–III)—concentrated in §7's 3 verbose models, vs. 0.38% (≤ 4.13%/task) across the other 17 (11/20 are 0.00% on both N/A and Old≠New; ablation N/A rates in ALLy Q2).
  • Task IV (model & judge sides): With no option extraction, 0.40% (51/12,600) of Table 2 model outputs are empty (0.00%), refusals (0.21%, 27 GPT-4o), or cut off before explanation (0.19%: 23 Qwen3-VL-2B, 1 GLM-4.6V-9B-Flash; 17/20 models 0.00%, ablations ≤ 1.59%), all scored as given; judge JSON cutoff drops from 0.47% (0.33% Table 2) to 0.00% (§8).

7. Updated Table 2 results, Overall re-standardization & updated findings (§1, §4.2, §4.3, Table 21b, Appendix A.11). Task I is unchanged for all 17 open-weight models (GPT-4o F1 +0.3), and the 250-pair control (Part 1) is 100% identical. Substantial MCQ shifts appear only on the 3 verbose models:

ModelOld≠New (II/III%)Task II (Onto/Caus)Task III (Onto/Caus)Overall (Rank)
Qwen3.5-35B-A3B77.6 / 67.821.9/1.3 → 49.2/52.432.7/25.3 → 76.3/80.441.6 (18) → 57.9 (4)
GLM-4.6V-106B-FP863.8 / 67.931.5/21.9 → 43.7/57.424.9/18.0 → 80.5/76.938.7 (19) → 52.8 (9)
Qwen3.5-9B56.0 / 21.634.2/30.9 → 42.4/52.474.9/67.1 → 84.6/78.651.3 (10) → 55.5 (6)
GLM-4.6V-9B-Flash0.8 / 4.038.8/47.3 → 38.8/48.778.0/67.4 → 80.3/70.647.8 (12) → 46.0 (14)
  • Why Overall shifts slightly elsewhere (GLM-4.6V-9B-Flash 47.8 → 46.0): Per §3.3 / A.15, Overall is a 20-model z-score (50 + 10 z̄). Updating the 3 verbose models raises pool μ_t (37.9/46.0 → 40.2/51.5 T-II; 73.3/65.8 → 78.8/72.2 T-III) and narrows σ_t. Excluding those 3 models, top-4 (Qwen3.5-27B, Gemini-3-Flash, Qwen3-VL-32B, Qwen3.5-122B) are unchanged and the 17-model order holds up to ≤ 0.4-pt near-ties (ρ = 0.99).
  • Retained vs. updated claims: Retained: (i) T-II (40.2/51.5) stays far below T-III (78.8/72.2), and on the 100-item human subsets all models trail humans significantly on Task IV (≤ 71.2 vs. 81.4 re-judged, 20/20 paired 95% CIs exclude zero; #1 trails by 6.0 ACC / 4.9 F1 pts on Task III—paired CI spans zero for #1, excludes zero for 19/20 on ACC & 18/20 on F1, ALLy W3); (ii) 250-pair control holds (Part 1); (iii) temporal sensitivity holds (Table 23). Updated in text: (i) §4.2 (Finding 1): remove the Qwen3.5-35B-A3B "low MCQ vs. high Task IV" example; (ii) §4.3 (Finding 5, Table 17) & A.11 (Table 22): update GLM-4.6V-106B T-II/III Thinking (80.5/76.9 vs. 79.2/75.4) and 64f (57.4 → 56.8), focusing Finding 5 on Thinking gains for Qwen3.5-122B (T-II–IV) and GLM-4.6V-106B (T-IV); (iii) Table 21b (multi-anomaly): standardize all three models to a unified 16f Instruct (4,096-token) protocol (N/A ≤ 2.5%).

8. Formalized Task IV rubric & judge token-budget update (Appendix A.4, A.14, A.17). In the submitted judge run, 0.47% of calls (0.33% in Table 2) reached the output-token cap before emitting JSON and scored 0. We re-scored all model and human responses at 16,384 max tokens under our formalized rubric and synchronized references (ALLy W1–W3: updating A.14 Ex. 3, stating the 15 + 20 split, anchoring reasoning to the reference anomaly). All calls now parse (0.00% fallback zeros); the 20-model mean moves from 61.30 (submitted) to 60.18 (revised; 60.14 before syncing 6 reference strings, ALLy W1), with 0/20 Overall rank changes vs. re-extracted Table 2, higher cross-judge agreement (ρ = 0.961 → 0.985, max shift 2), and smaller OpenAI-family residual bias (−3.77 → −1.13 pts).

9. Revision & v2 release. All manuscript updates are marked in blue, and our v2 release at huggingface.co/datasets/humannobody/video-hoca (and humannobody/task1-250-pairs) provides: (i) raw outputs & extractions, (ii) judge scores/sub-scores & justifications, (iii) static_dynamic_labels.jsonl & synchronized ground_truth.jsonl, and (iv) an all-clip license_manifest.jsonl.

AuthorsOpenReview ↗

Official Comment — Summary of Revisions (all updates marked in blue in the revised PDF)

Table and appendix numbers below follow the revised PDF (submitted Tables 3, 4, 6, 7, 8, 10b, 11, and 12 are now Tables 5, 4, 11, 15, 17, 21b, 22, and 23):

  • Generator-matched control promoted to main text (§4.2, Tables 4–6 & Abstract): Moved the 250-pair control tables (Tables 4–5) into §4.2, added an item-matched normal/anomalous decomposition holding the same 250 anomalous clips and their predictions fixed (Table 6; 72.7% → 64.4%, Δ = −8.3, normal-half 91.4% → 74.8%, anomaly recall 54.1%) and the 500-clip padding audit (ΔACC = +0.02%), and shifted secondary §4.4 prose to Appendix A.4/A.5. (ALLy W4, RB9G Minor 2, YdBB W1/Q3)
  • Option extraction alignment & N/A reporting (§4.1–§4.3, Table 2, Tables 17/21b/22/23, Appendix A.16 Tables 28–29): Replaced the initial regex with our blind GPT-OSS-120B + strict deterministic template pipeline (360/360 human verification agreement; Table 2 N/A ≤ 1.43%), updated all affected tables and Abstract/§4.2 means, removed the Qwen3.5-35B-A3B Finding 1 example, and focused Finding 5 (§4.3) and Appendix A.11 on verified Thinking/frame effects with item-paired bootstrap 95% CIs. (YdBB Q2, ALLy W5/Q2/Q3, RB9G W2/Q4)
  • Formalized Task IV rubric & sub-scores (§4.2, Appendix A.4 Tables 11–14, A.13 Tables 25–26, A.14, A.17): Updated A.14 Example 3 ("Causal", marked synthetic) with L2/L3 tags and L2→L1 mapping, synchronized the Task IV Process Type reference strings in ground_truth.jsonl with the L2→L1 mapping (630/630 consistent), raised judge max_tokens to 16,384 (0.00% JSON cutoffs; 20-model mean 60.18, 0/20 Overall rank changes relative to the re-extracted Table 2 baseline), added explicit 4a (/15) + 4b (/20) reasoning sub-scores to Table 11, and added the 150-item controlled explanation experiment to Appendix A.4 (Table 12). (ALLy W1/W2, YdBB W2)
  • Human baseline protocol & same-item CIs (§3.3, §4.2, Table 2, Ethics Statement): Clarified the 100-video stratified subsets (2 independent evaluators per task), aligned Task III human branch macro-F1 (95.3%, 94.1/96.1 Onto/Caus; 96.0% ACC), re-scored human Task IV responses under the finalized rubric and synchronized references (81.43 / 81.39), reported same-item model comparisons with paired 95% CIs (20/20 exclude zero on Task IV; 19/20 ACC / 18/20 macro-F1 on Task III), and added recruitment/compensation/consent details. (ALLy W3, RB9G W1/Minor 3/Q1/Q2)
  • Reporting granularity, blind baseline, Task III visual rebuild & normal-video abstention (§3.1–§3.3, §4.2, §5, Tables 2–3, Appendix A.3 Tables 8–10, A.9 Table 19, A.10 Table 20, A.12 Table 24): Clarified L1/L2/L3 roles and "Fine-grained", added Table 2 bootstrap 95% CIs (Table 9), a symmetric inter-annotator L2 pairwise-count matrix from three external annotators and a row-normalized model L2 confusion matrix (Table 8), and video-bootstrap category profiles (Table 10), added the 4-task blind baseline (Table 2) and 630-item visually grounded Task III 2×2 study (Table 3 & Table 19), added static/dynamic stratified CIs (Table 24), and added the 250-clip "No anomaly" abstention evaluation (Table 20). (YdBB W1/W2/Q1/Q3, ALLy W1/Q3)
  • Related work & v2 artifact release (§2, Table 1, repositories): Added [1–4] along generator-realism vs. VLM-reasoning tracks in §2 and Table 1, and released raw predictions/extractions, judge sub-scores, static_dynamic_labels.jsonl, and the 1,689-clip license_manifest.jsonl in v2. (RB9G Strengths/Minor 1/Q3, ALLy Q4, YdBB W2/Q3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.