- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether observed benchmark accuracy reliably reflects visual understanding capability in video anomaly reasoning. The authors propose a capability decomposition audit framework that examines three sources of evaluation confounding: non-visual shortcuts, visual ranking ability, and decision operating-point bias. The framework introduces residual side-channel capacity to quantify the additional label information available from unintended metadata, employs Bias-Invariant Structured Inference (BISI) to analyze decision biases under structural constraints, and explores rank-based selective prediction through RankGuard. Experiments across video anomaly benchmarks reveal substantial metadata leakage and demonstrate that apparent performance improvements can arise from decision shifts rather than improved visual discrimination. The paper further evaluates the limitations of these interventions and provides reproducible auditing tools.
Critical strengths
The paper addresses a meaningful issue in video anomaly understanding: high benchmark accuracy does not necessarily imply strong visual understanding. The findings on metadata leakage and decision-threshold shifts provide valuable evidence that existing evaluation protocols may overestimate or mischaracterize model capabilities.
Critical weaknesses
-
The main technical components are largely built upon established concepts, including conditional mutual information, constrained decoding, and selective classification. Although their application to video anomaly benchmark auditing is meaningful, the paper does not sufficiently demonstrate a fundamental methodological advance over existing evaluation and calibration approaches.
-
Several important conclusions rely on relatively small subsets or dataset-specific artifacts. In particular, the rule-based anomaly analysis involves only 24 samples, while the strongest shortcut results originate from specific metadata or question-order correlations. It remains unclear whether the proposed findings and diagnostic framework generalize to broader video anomaly benchmarks and diverse model architectures. This limits the strength of the paper's broader claims about visual capability evaluation.
Questions for authors
-
What are the fundamental methodological contributions of the proposed framework beyond existing information-theoretic auditing, constrained inference, and selective prediction techniques? Could the authors clarify which diagnostic capabilities cannot be achieved by directly combining established methods, and provide stronger comparisons against existing evaluation approaches?
-
How robust are the main findings across broader benchmarks and model architectures? Given the limited size of the rule-based anomaly subset and the dataset-specific nature of several shortcut findings, could the authors provide additional cross-dataset or cross-model evidence demonstrating that the identified confounding effects are systematic rather than isolated to particular benchmark construction artifacts?