acceptodds
Withdrawn from ICLR 2027

Observed Accuracy Is Not Visual Capability: Auditing and Repairing Decision-Level Confounds in Video Anomaly Benchmarks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A rule reading only the video header, meaning resolution, duration and frame count, predicts a video anomaly benchmark’s labels at 0.970. We had adopted that benchmark because it looked clean. How much of such a shortcut survives a strong model? We formalize the question as residual side-channel capacity, the drop in optimal predictive risk from adding an unintended channel to a frozen model score, which under log loss is I(Y ; Z | S); our plug-in estimate is a lower bound on it. Above the strongest model we measure, video headers close 42% of the label uncertainty its score leaves open, lifting AUC from 0.947 to 0.982. On a rendered benchmark the same audit returns capacity indistinguishable from zero, so this is a property of how footage was collected, not of the estimator. A benchmark score is an intermediate observable, and reading capability off it means first showing which layer moved. We therefore separate what one score conflates, acquisition shortcuts, visual ranking and the decision operating point, probing each by its own intervention and releasing the audit as a harness with cached scores that reproduces every headline number on CPU in seconds. Task structure makes part of any score unidentifiable, which gives an exact account of when repairing an operating point helps, hurts or does nothing. Because ranking transfers across domains and absolute scale does not, a certified rank-based selective rule retains 1.2–1.7× the coverage of an equally certified score-threshold rule at matched risk, with paired intervals excluding zero in three of four settings. We also report four revisions to our own conclusions and 18 pre-registered negative verdicts.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
benchmark auditing, video anomaly detection, shortcut learning, dataset bias, selective prediction, evaluation methodology, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rAVpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether observed benchmark accuracy reliably reflects visual understanding capability in video anomaly reasoning. The authors propose a capability decomposition audit framework that examines three sources of evaluation confounding: non-visual shortcuts, visual ranking ability, and decision operating-point bias. The framework introduces residual side-channel capacity to quantify the additional label information available from unintended metadata, employs Bias-Invariant Structured Inference (BISI) to analyze decision biases under structural constraints, and explores rank-based selective prediction through RankGuard. Experiments across video anomaly benchmarks reveal substantial metadata leakage and demonstrate that apparent performance improvements can arise from decision shifts rather than improved visual discrimination. The paper further evaluates the limitations of these interventions and provides reproducible auditing tools.

Critical strengths

The paper addresses a meaningful issue in video anomaly understanding: high benchmark accuracy does not necessarily imply strong visual understanding. The findings on metadata leakage and decision-threshold shifts provide valuable evidence that existing evaluation protocols may overestimate or mischaracterize model capabilities.

Critical weaknesses
  1. The main technical components are largely built upon established concepts, including conditional mutual information, constrained decoding, and selective classification. Although their application to video anomaly benchmark auditing is meaningful, the paper does not sufficiently demonstrate a fundamental methodological advance over existing evaluation and calibration approaches.

  2. Several important conclusions rely on relatively small subsets or dataset-specific artifacts. In particular, the rule-based anomaly analysis involves only 24 samples, while the strongest shortcut results originate from specific metadata or question-order correlations. It remains unclear whether the proposed findings and diagnostic framework generalize to broader video anomaly benchmarks and diverse model architectures. This limits the strength of the paper's broader claims about visual capability evaluation.

Questions for authors
  1. What are the fundamental methodological contributions of the proposed framework beyond existing information-theoretic auditing, constrained inference, and selective prediction techniques? Could the authors clarify which diagnostic capabilities cannot be achieved by directly combining established methods, and provide stronger comparisons against existing evaluation approaches?

  2. How robust are the main findings across broader benchmarks and model architectures? Given the limited size of the rule-based anomaly subset and the dataset-specific nature of several shortcut findings, could the authors provide additional cross-dataset or cross-model evidence demonstrating that the identified confounding effects are systematic rather than isolated to particular benchmark construction artifacts?

Reviewer_uKTHOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what accuracy scores actually measure across five video anomaly benchmarks. The authors first show that Qwen-VL models handle obvious anomalies like crashes well, but struggle with traffic rule violations. Scaling visual tokens from 1x to 8x seems to boost recall, but only because the model predicts "Abnormal" more often, and recall at 10% FPR stays flat. Next, they show that several benchmarks have strong shortcuts in question wording or video metadata (like resolution and duration) that predict labels without looking at the frames. Finally, they propose BISI and RankGuard for test-time calibration, though top-K decoding fails to beat simple threshold tuning on accuracy and only helps with selective abstention.

Critical strengths

The finding on visual token scaling in Section 5 is a good catch. Using 8x more visual tokens only makes the model predict "Abnormal" more often, while recall at 10% FPR stays flat at 0.21 and AUC stops improving after 2x.

The shortcut analysis in Section 6 is very practical. Simple classifiers using only video resolution, duration, or question wording reach 90% to 97% accuracy on benchmarks like Roundabout and AIC Track 3 without looking at the video frames.

Testing the synthetic UBnormal dataset as a negative control is a nice touch. Metadata features stay near chance (0.53 AUC) on UBnormal, which confirms that the shortcuts on the real datasets come from video collection artifacts rather than probe overfitting.

Critical weaknesses

The paper has way too many formulas and theorems for how few experiments it runs. Section 7 turns basic linear algebra and simple counting of threshold flips into five formal theorems, while the experiments only test three 7B/8B Qwen-VL models and 24 rule-based videos (with 18 failed variants in Table 12).

As a post-hoc calibration method, BISI and RankGuard do not beat simple baselines. In Table 2, transductive top-K needs the whole test batch at once yet loses to a majority guess and simple thresholding on Roundabout, while RankGuard abstains on about 90% of samples at B = 64 (eps = 0.10).

The main claims in Sections 4 and 5 rely on a tiny sample size and one model family. There are only 24 rule-based videos in VAU-Bench (so 0.29 vs 0.21 recall is just two videos), and only 7B/8B Qwen-VL models are tested.

There are two statistical issues in the RSCC section. The Abstract claims the cross-fitted log-loss drop is a strict lower bound on I(Y; Z | S), which is not generally true for the difference of two variational upper bounds, and rscc.py splits folds and bootstraps by clip instead of by source video.

The writing is hard to follow and reads like rough working notes. Terms like BISI and RankGuard appear on page 2 before they are defined, the main model is unnamed in Section 3, dataset names change across sections, and too much space is spent on failed ideas and retracted drafts.

Questions for authors

In the supplementary code, rankguard.py runs the 30-point grid search on the certification split instead of the single-test protocol in Appendix P. Did you upload an older version of the script by mistake, and how often does RankGuard output zero coverage across splits?

How do BISI and RankGuard compare against standard post-hoc calibration (like Platt or temperature scaling) and recent rule-based video anomaly methods like AnomalyRuler?

How much do the RSCC results and confidence intervals in Table 1 change if you group the cross-validation folds and bootstrap by source video instead of splitting by clip?

Can you fill in the missing B = 16 and B = 32 rows in Table 3, report how many clips remain after 10-bin matching in Table 1, and test at least one non-Qwen VLM family on the 1x to 8x token scaling experiment?

Reviewer_5TC9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an auditing protocol for VAD. The writing and formatting of the paper, including the introduction, methodology and experimental sections, are too dense, opaque and vague to fully understand due to the heavy use of LLMs.

Critical strengths

N/A

Critical weaknesses

The paper proposes an auditing protocol for VAD and buries it in the Appendix, Section B. It also audits one specific checkpoint of one model (L144) on a subset of its own dataset and a subset of another dataset (AIC Track 3 TAR, a transportation reasoning benchmark). The authors do not justify any of these choices. The paper is badly written, dense, confusing and unnecessarly long-winded. It contains no informative figures, comparisons with other approaches, and informal language that is not fit for a scientific publication (L427: "Given B target labels, why not fit a threshold? We ran it with the B labels as calibration, the rest untouched test, 200 splits; the answer surprised us."). The paper is clearly, mainly the product of an LLM agent. Even if the paper contained novel, important insights, its current form is not fit for publication to the point of being almost not fixable in the rebuttal phase.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.