acceptodds
Under review as a conference paper at ICLR 2027

What Video Anomaly Probes Measure

Abstract

An above-chance probe score on frozen video features is often read as evidence that the representation captures anomaly. We argue that this reading needs an evaluation contract, and we audit the supervised probe with four low-dimensional controls—scene complexity, video identity, position in video, and detected-agent counts—under scene-grouped folds, paired rows and video-clustered uncertainty, on a frozen V-JEPA 2 encoder over UCF-Crime and NWPU-Campus. On UCF-Crime, a linear probe given only the first encoder window of each clip predicts whether the clip will contain an anomaly at video-level AUROC 0.899; this is a video-level diagnostic, not evidence of temporal anticipation. On NWPU-Campus, a nine-dimensional count vector is a strong comparator under the diagnostic: with each arm's better of two heads, no tested representation arm is superior after paired Holm correction, and the count arm is the highest selected arm in every one of 25 stored seeds. That ordering is conditional on the readout: under a fixed linear head the count arm sits at chance and no arm is distinguishable from it, so we report paired tests within each head family rather than a single ranking. The seed sweep jointly varies fold assignment and optimisation and moves arms by up to 0.186 AUROC, widest for the complexity control. Under the benchmark's own normal-only protocol the same counts reach at most 60.6 micro-AUC against a published 76.9, whereas the frozen state reaches 59.1 micro-AUC and an exploratory 67.2 macro-AUC in the prespecified estimator cell: a control can dominate a supervised diagnostic without being a competitive detector. Two analytical results—that a prediction-error residual is a dispersion functional, and that a self-normalised pre-hoc hazard measures the model's own tail mass—motivate the complexity control and explain why no unsupervised anticipation arm appears. We release the protocol and every artefact, and limit the conclusion to measurement design: an anomaly score should state its target, information cutoff, controls, grouping, readout selection, aggregation and uncertainty before it is read as evidence about a representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video anomaly detection, linear probing, evaluation methodology, confounds, video world models, V-JEPA 2, benchmark audit, self-supervised video representations, negative results

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kJYQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a critical methodological concern in video anomaly detection research: whether above-chance probe scores on frozen video features genuinely indicate that the representation captures anomaly semantics. The authors propose a benchmark-audit protocol that incorporates four low-dimensional controls (scene complexity, video identity, position in video, and detected-agent counts) with scene-grouped folds, paired rows, and video-clustered uncertainty. The protocol is applied to a frozen V-JEPA 2 encoder over UCF-Crime and NWPU-Campus datasets. Key contributions include: (i) a reusable control protocol for diagnostic probes, (ii) evidence that simple controls, readout selection, and aggregation materially change conclusions on two surveillance corpora, and (iii) an artifact-backed reporting discipline that separates diagnostic probing from native anomaly detection and temporal anticipation. The paper's central finding is that a pre-event video-level diagnostic reaching AUROC 0.899 on UCF-Crime reflects video identity rather than temporal anticipation, and that object counts serve as a strong comparator under certain readouts yet fail under the benchmark's native protocol.

Critical strengths
  1. The paper carefully separates diagnostic probing, native anomaly detection, and anticipation, which are often conflated in the literature.
  2. The paper addresses a fundamental issue in how the community interprets probe scores, which has implications for a broad range of video understanding research.
  3. The finding that a linear probe on the first encoder window achieves AUROC 0.899 on UCF-Crime, and that object counts can rival learned representations under specific readouts, provides valuable cautionary evidence.
Critical weaknesses
  1. The evaluation is restricted to only two surveillance datasets (UCF-Crime, NWPU-Campus) and a single frozen encoder (V-JEPA 2). This narrow scope limits the generalizability of the conclusions.
  2. The paper does not include comparisons with recent state-of-the-art anomaly detection methods, such as those based on large language models, diffusion models, or other self-supervised approaches
  3. The paper does not deeply analyze why the frozen representation fails to beat control variables, nor does it investigate how different layers of the encoder, different pooling strategies, or fine-tuning might change the results.
Questions for authors
  1. How do you reconcile the claim that video identity is a strong predictor with the practical AD use case where models must generalize to new scenes?
  2. Have you tested whether other recent video encoders (e.g., VideoMAE, InternVideo, etc.) show different patterns? Would your conclusions hold for contrastive or reconstruction-based encoders?
  3. Could you provide more analytical discussion of what kinds of anomaly signals are preserved vs. removed?
  4. How sensitive are the conclusions to the choice of anomaly score aggregation?
  5. How do you address the concern that your control variables (e.g., object counts) may themselves be legitimate anomaly signals in real surveillance scenarios?
Reviewer_fH8UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

(1) This paper studies what supervised probes on frozen video representations measure in video anomaly detection.

(2) The authors propose an evaluation protocol with four controls: scene complexity, video identity, position in video, and object counts.

(3) Experiments with V-JEPA 2 on UCF-Crime and NWPU-Campus show that pre-event features can predict video-level labels and that probe rankings depend on the readout and evaluation protocol.

Critical strengths

(1) The paper addresses an important evaluation problem.

(2) The paper is well organized and easy to follow.

(3) The authors conduct extensive experiments and provide detailed analyses.

Critical weaknesses

Introduction:

(1) The authors state "Two analytical results sit behind the protocol" in Section 1. These results concern prediction-error residuals and self-normalised pre-hoc hazards, whereas the main experiments evaluate supervised probe scores. The connection between these analytical results and the probe experiments is not established.

(2) The authors state "which is why a scene-complexity control belongs in the protocol" in Section 1. However, the complexity-control AUROC in Tables 1 and 2 measures label predictability; it does not directly test whether prediction residuals vary with scene complexity or explain the probe rankings.

Related Work:

(1) The related work omits directly relevant studies, including but not limited to [1, 2]. The authors need to clarify the novelty and necessity of the proposed evaluation protocol.

[1] Georgescu M I, Ionescu R T, Khan F S, et al. A background-agnostic framework with adversarial training for abnormal event detection in video[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 44(9): 4505-4523.

[2] Ramachandra B, Jones M. Street scene: A new dataset and evaluation protocol for video anomaly detection[C]//Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2020: 2569-2578.

Method:

(1) The authors state "What it takes to claim anticipation is a within-video design" in Section 4.1. However, Section 3 does not incorporate the near-onset versus far-from-onset comparison into its central protocol or specify how this comparison should be combined with the four controls to assess an anticipation claim.

(2) The authors state "the other three are empirical" when explaining the choice of controls in Section 3. I wonder what theoretical or empirical criteria justify selecting these four controls, whether they overlap or are correlated, and what distinct diagnostic information each contributes. Their complementarity and the coverage of alternative sources of label predictability remain unclear.

Experiments:

(1) The authors state "one frozen video encoder, two surveillance corpora" when defining the scope in Section 2. The principal comparisons use V-JEPA 2 features with shallow probes, leaving the applicability of the proposed protocol to other video representations and anomaly detection methods insufficiently validated.

(2) The authors state "on two surveillance corpora" in Section 1. However, pre-event video discrimination is primarily demonstrated on UCF-Crime, whereas the count comparisons, readout comparisons, and seed sweep are primarily conducted on NWPU-Campus. The consistency of the full control protocol and its main findings across datasets remains insufficiently validated.

(3) The authors state "both with the same epochs and seed" when describing the two probe heads in Section 3. However, the main paper does not specify the epoch count, learning rates, batch sizes, optimizer, or MLP hidden width, leaving the training procedure insufficiently specified.

(4) The authors state "The concatenation arms (behaviour + counts, ROI + counts) score below counts alone under both heads" in Section 4.2. Table 1 shows that their adjusted MLP AUROCs fall from 0.686 for counts alone to 0.556 and 0.478. Although the authors acknowledge the optimization issue, the reported experiments do not establish whether the count advantage survives validated training settings for the different inputs.

(5) The authors state "Probe folds are grouped by scene on NWPU-Campus" in Section 3. However, Appendix E.4 states that folds are grouped by video while describing the setup as "the protocol of Section 3 throughout". The reported grouping unit therefore differs from the protocol it claims to follow.

Questions for authors

Please see the Critical Weaknesses section.

Reviewer_gXB6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to audit probing in VAD. However, the paper is written so densly and vaguely that it is very difficult to follow and fully understand.

Critical strengths

N/A

Critical weaknesses

This paper is supposedly an audit of probing techniques on video anomaly detection. It appears to provide no novel insight, no comparison with existing methods, no definitive results. It is badly written, hard to follow, offers no visualizations that could clarify any of the aspects that it supposedly tackles, and contains basic mistakes (i.e. "We fit a linear probe \textit{inside} the labelled UCF-Crime test split [...]" at L192). Anybody that has used Claude models (as disclosed by the authors) could recognize the patterns:

  • A subsection titled "UCF-Crime: the trap, and it is the paper’s own thesis." in L1246 (and similarly at L1270 and L413)
  • 54 instances of "rather than", 23 instances of "[...], not [...]", 68 instances of "arm".
  • Tables 1, 2 and 3 are not in the style of research papers. Table 3 contains a column with no name, and the values in its cells are text: "loses" or "n.s.", the latter is not even defined anywhere in the text.

I do not believe that the paper can go from the current state to a state fit for publication in ICLR within the rebuttal period.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.