acceptodds
Under review as a conference paper at ICLR 2027

Predictive Transitions for Video Anomaly Detection

Abstract

We derive a direct anomaly signal from a representation learned to condition future-frame generation, using a state–transition framework for normal-only video anomaly detection. Current appearance and ordered frame and feature changes jointly condition a diffusion Transformer. After generative training, we freeze the encoders and fit a small head to predict the next local frame change from transition features. Matched representation probes on Ped2, Avenue, and ShanghaiTech show that learned transition features outperform raw frame-change history for anomaly detection. Interventions reveal dependence on transition content and temporal order, with clearer temporal-order effects on Ped2 and Avenue. Together, the probes and conditioning ablations show that transition features can support anomaly detection even when hybrid conditioning does not improve frame-error AUC. The change score can be computed from the learned encoders and prediction head without iterative frame generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video anomaly detection, normal-only anomaly detection, representation probing, future-frame prediction, diffusion transformer, flow matching, temporal representation, conditional generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ooEdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a state–transition framework for normal-only video anomaly detection. The observed video history is decomposed into a state representation, which captures the current appearance, and a transition representation, which encodes ordered frame and feature differences. These representations jointly condition a diffusion Transformer trained for future-frame generation. After the generative model is trained, the encoders are frozen and a lightweight change-prediction head is trained on the transition representation to predict the next local frame change. The resulting change-prediction error is used as an anomaly score, either alone or in combination with the conventional future-frame prediction error.

The paper evaluates the learned transition representation using frozen-representation probes and inference-time interventions. The experiments compare transition features with state features, feature differences, and raw frame-change history, and further examine the effects of perturbing transition content and temporal order. The method is evaluated on Ped2, Avenue, and ShanghaiTech.

Critical strengths
  1. The paper investigates transition representations separately from conventional frame-prediction error. An interesting aspect of the paper is that it does not evaluate transition information only through its contribution to future-frame reconstruction/prediction. Instead, the authors directly probe the learned transition representation through a future-change prediction task. This helps distinguish whether temporal information is present in the representation even when it does not improve frame-error-based anomaly detection.

  2. The representation-probing experiments are useful and reasonably controlled. Table 1 applies the same change-head architecture to several frozen representations, including the current state, the last feature difference, raw frame-change history, and the learned transition representation. Learned transition features consistently outperform raw frame-change history across all three datasets, although the margins are relatively modest.

  3. The temporal-order interventions provide some evidence that the learned representation captures ordered temporal information. Reversing or cyclically shifting the transition history reduces change-based anomaly detection performance, especially on Ped2 and Avenue. This is a useful diagnostic experiment showing that the transition encoder does not merely encode the overall magnitude of motion/change.

Critical weaknesses
  1. The paper does not establish that explicitly separating a transition representation provides a meaningful advantage over conventional multi-frame temporal modeling. Existing future-frame prediction methods already take multiple frames as input, and sufficiently expressive temporal encoders can implicitly learn motion and temporal dynamics. The key missing comparison is therefore a capacity-matched model that directly encodes the same sequence of raw frames without explicitly constructing frame/feature differences. Table 1 only compares the proposed transition representation with state features, feature differences, and raw frame-change history. As a result, the paper shows that its learned transition feature is better than raw differences, but does not demonstrate that explicit transition modeling itself is necessary or superior to standard multi-frame encoding.

  2. The necessity of the generative training framework is unclear. The transition encoder is first trained through conditional future-frame generation, then frozen, and finally used for future-change prediction. However, there is no direct baseline where the transition encoder and change head are trained end-to-end for next-change prediction. If such a simpler model achieves similar performance, the diffusion Transformer, flow-matching objective, and two-stage training procedure may not be essential. The paper itself acknowledges that the necessity of generative training remains unresolved.

  3. The final reported performance is difficult to attribute to the proposed transition representation because of substantial dataset-specific tuning and additional score engineering. The score transformation, frame/change mixing weight lambdalambda, and median-filter length are selected using test AUC, and the selected values differ substantially across datasets. In addition, the change target and anomaly score involve several designed components, including signed pooling, magnitude components, active/static loss balancing, and top-kappakappa aggregation. Without sufficient ablations, it is difficult to determine how much of the gain comes from the transition representation itself versus these target, scoring, and post-processing choices.

  4. Several experiments needed to support the central claims are missing or insufficiently robust. The method always uses six observed frames, despite the claim that ordered transition history is important, so an ablation over history length would be valuable. The benefit of transition conditioning is also inconsistent across datasets: State + Transition improves over State on Ped2, but is worse on Avenue and nearly identical on ShanghaiTech. Moreover, ShanghaiTech uses only one backbone seed even though several reported differences are below one percentage point. Finally, the comparison with published methods is not strictly controlled because the proposed method reports video-macro AUC while prior results retain their original evaluation protocols.

Questions for authors
  1. Can the authors provide a capacity-matched baseline that directly encodes the same sequence of raw frames without explicitly constructing frame or feature differences? This comparison seems necessary to establish whether explicit transition modeling provides an advantage over conventional multi-frame temporal encoding.

  2. How does the proposed transition encoder compare with a standard temporal encoder, such as a Transformer operating directly on per-frame features, under the same architecture size and training objective?

  3. How does the proposed generatively pretrained transition encoder compare with one trained directly and end-to-end for future-change prediction? This would help determine whether future-frame generation is necessary for learning the useful transition representation.

  4. What happens if the diffusion/flow-matching pretraining stage is removed entirely and only the change-prediction objective is used?

  5. Why are the score transformation, lambdalambda, and median-filter length selected based on test AUC? Can the authors report results using only validation-selected hyperparameters?

  6. Can the authors report a single common scoring/post-processing configuration across all three datasets? This would help assess how much performance depends on dataset-specific tuning.

  7. How sensitive are the final results to lambdalambda and the median-filter length? A sensitivity plot or table would help clarify whether the reported values correspond to broad stable regions or narrow optima.

  8. Can the authors provide ablations for the future-change target design? In particular, what are the individual contributions of positive/negative signed pooling and the average/max magnitude components?

  9. How much does active/static cell loss balancing contribute to the change-prediction performance?

  10. How much does the top-kappakappa aggregation contribute compared with simply averaging all local errors?

  11. How does performance change with the number of observed frames? For example, results for 2, 4, 6, and 8 frames would help determine whether modeling a longer ordered transition history is actually beneficial.

  12. How does the method compare when only the most recent transition is used instead of the full transition history? This would help separate the benefit of ordered history from the benefit of simply using frame differences.

  13. Why does State + Transition conditioning underperform State-only conditioning on Avenue? Does this indicate that the transition condition may sometimes interfere with the future-frame generator?

  14. Can the authors explain why transition conditioning contributes strongly on Ped2 but very little on Avenue and ShanghaiTech? Is this related to scene complexity, motion statistics, or optimization?

  15. Can the authors report multiple-seed results on ShanghaiTech? Several differences involving the transition representation are below one percentage point.

  16. Can the authors evaluate the proposed method and key baselines under exactly the same AUC protocol? Since Table 5 combines the paper's video-macro AUC with previously published numbers using their original protocols, it is difficult to interpret the comparison quantitatively.

  17. Can the authors clarify the intended primary contribution of the paper? Is the main contribution the state–transition generative architecture, the learned transition representation, or the change-based anomaly scoring mechanism? The current experiments provide different levels of support for these components.

  18. Why is the evaluation limited to Ped2, Avenue, and ShanghaiTech? Could the authors provide results on more recent and diverse VAD benchmarks, such as UBnormal or other modern datasets? Given that the proposed method explicitly models temporal transitions, evaluation on datasets with more diverse scenes, anomaly types, and motion patterns would help assess whether the observed benefits generalize beyond the traditional benchmarks used in this paper.

4 comments

Thank you for the detailed questions. We completed controls separating representation, training objective, aggregation, and evaluation protocol.

  1. Raw temporal encoding and direct training

Raw-sequence and explicit-difference encoders share widths, temporal/spatial blocks, and the change head. Parameter counts differ by 256 parameters. Raw inputs have six positions and differences have five; capacity is closely matched, while FLOPs differ.

A: Raw/direct change prediction. B: Raw/generative pretraining, then frozen-encoder change-head training. C: Differences/direct change prediction. D: Differences/generative pretraining, then frozen-encoder change-head training.

Pooled AUC (%), mean ± sample SD over three training seeds:

VariantPed2AvenueShanghaiTech
A89.81 ± 0.2285.30 ± 0.0268.66 ± 0.27
B92.34 ± 0.3782.21 ± 0.1968.88 ± 0.13
C88.14 ± 0.7785.82 ± 0.0968.74 ± 0.12
D91.87 ± 0.4482.42 ± 0.1168.63 ± 0.09

D-B compares encodings under pretraining: -0.48, +0.21, and -0.25 percentage points. D-C compares pretraining with direct training: +3.73, -3.40, and -0.11 points. Encoding performance is closely related; pretraining benefits Ped2, while direct training performs strongly on Avenue.

Direct models receive 10,000 updates; pretrained models receive 5,000 generative plus 5,000 head updates. Targets, head architecture, loss, and scoring are shared. This matches updates, not compute or convergence.

  1. Common evaluation and sensitivity

The new study uses video-disjoint normal train/validation splits, validation-loss checkpoint selection, and a common score without fusion, smoothing, or whole-video normalization. Pooled AUC uses all scored frames; macro AUC averages videos containing both classes.

Historical Ped2 macro AUC reproduces 99.657% under test-selected post-processing. New D runs obtain 93.76 ± 0.19% under the fixed protocol. These are separate training runs on previously examined benchmarks.

A retrospective sensitivity audit covers 196 configurations per dataset, varying fusion, median windows, normalization, and filter direction. Historical Ped2 change scores yield 95.755% macro and 92.865% pooled AUC without smoothing or whole-video normalization. This diagnostic is separate from validation-based selection.

  1. Target, loss, and aggregation

For the same D models, three-seed mean pooled AUC is:

AggregationPed2AvenueShanghaiTech
Mean local error77.6379.8266.45
Mean + mean of top-four errors91.8782.4268.63

Aggregation contributes substantially, particularly on Ped2, and is shared across A-D. Component-wise retraining ablations for signed pooling, average/max magnitude, and active/static loss balancing remain to be completed.

  1. History and conditioning

Order interventions demonstrate temporal-order sensitivity; the last-feature-difference probe provides a short-history reference. Matched retraining for 2/4/6/8 frames and a single-transition encoder remains to be completed.

Conditioning interventions characterize dataset-specific behavior. Scene complexity, motion statistics, and optimization remain candidate explanations rather than isolated causes.

  1. Seeds and external evaluation

The A-D study includes three training seeds on every dataset, including ShanghaiTech.

Local MA-PDM evaluations use identical scored-frame cohorts, labels, and metrics without smoothing or whole-video normalization. Its pooled/macro AUC is 86.73/87.52% on Avenue and 65.49/78.74% on ShanghaiTech. Each uses one local checkpoint with method-specific training and preprocessing, distinct from published benchmark results. Matched FFP and FPDM comparisons remain to be completed.

  1. UBnormal and contribution

Earlier normal-only UBnormal experiments cover three backbone seeds, without smoothing or test-video normalization. Transition pooled AUC is 55.66 ± 0.02%, versus 54.89 ± 0.17% for actual-state features and 56.18% for raw change history. These extend the frozen-probe analysis and are distinct from the new A-D protocol.

Our contribution separates temporal information in learned representations, its utility for change-based detection, and generation-free inference cost. The controls clarify the dataset-dependent roles of encoding, training, and aggregation.

Reviewer_ooEdOpenReview ↗

Remaining Concerns After Rebuttal

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my questions, especially the comparisons between raw and difference-based encoding and between direct training and generative pretraining.

After reviewing the new results, I still have two main concerns.

1. Performance and evaluation protocol.

The original results appear to depend heavily on test-selected post-processing. For example, the Ped2 macro AUC drops from 99.657% to 95.755% without smoothing or whole-video normalization. The new fixed-protocol results are also substantially lower than the originally reported numbers, although I understand these come from separate training runs.

The aggregation ablation is also concerning. On Ped2, changing from mean local error to mean + top-four aggregation improves AUC from 77.63% to 91.87%. This suggests that a substantial part of the reported performance depends on the scoring design rather than the learned representation alone. While the revised evaluation protocol is an improvement, I am still not convinced that the original performance claims are sufficiently supported.

2. Motivation for the proposed framework.

The new A–D experiments are helpful, but they also raise questions about the necessity of the proposed design. Explicit difference encoding does not consistently outperform raw temporal encoding. Similarly, generative pretraining improves performance on Ped2 but performs considerably worse than direct change-prediction training on Avenue, with almost no difference on ShanghaiTech.

I do not expect the proposed method to outperform every baseline on every dataset. However, if raw temporal encoding performs similarly and direct change prediction can achieve comparable or better results, it is unclear what advantage the proposed state–transition generative framework provides over these simpler alternatives.

Overall, I appreciate the additional experiments and clarifications. They address several of my original questions, but I do not think they resolve the two main concerns above. I therefore maintain my original rating.

Your distinction between finding useful information in a representation and showing that the proposed framework is needed to learn it helped us revise the paper. On evaluation, Tables 1–2 (p. 6) now show both aggregation rules on the same checkpoints. For example, on Ped2 the pretraining gain over direct training with difference inputs is +3.73 points with mean-plus-top-four aggregation but only +0.72 points with mean alone. We therefore attribute the larger gain to the representation and its readout together, not to the representation alone.

On motivation, the contribution we retain is that a temporal representation learned to condition frame generation can also score future changes without running the generator at inference. This is a useful property of an already trained generative model, not a reason by itself to prefer generative training when change detection is the sole objective. The revised conclusion (p. 7) makes that distinction explicit: pretraining helps on Ped2 under the primary scoring rule, direct training is stronger on Avenue, and the difference is small on ShanghaiTech.

Reviewer_ooEdOpenReview ↗

Official Comment by Reviewer ooEd

Thank you for the detailed response and revisions. However, my main concerns remain unresolved.

The revised results raise further questions about the novelty and effectiveness of the proposed framework. If raw temporal encoding achieves comparable performance to the proposed difference-based transition representation, and direct training performs similarly to or better than generative pretraining, it is unclear what advantages the proposed state–transition framework provides over these simpler alternatives.

Furthermore, the detection performance appears to depend heavily on the aggregation strategy. Without the top-four aggregation, the performance drops substantially, and the mean-only results do not provide convincing evidence of an advantage over existing methods. I also noticed that the revised paper no longer includes the original comparison table with existing VAD methods. While I understand that differences in training and evaluation protocols make direct comparisons difficult, I believe that comparisons under a consistent evaluation protocol, or at least clearly qualified comparisons with existing methods, are important for assessing the effectiveness of the proposed approach.

Overall, although the revisions clarify the limitations of the method, I am still not convinced that the proposed state–transition framework offers a sufficiently novel or effective approach to video anomaly detection. I therefore maintain my original rating.

Reviewer_joG1OpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a state transition framework for video anomaly detection trained only on normal videos, without using anomalous video data. It separately encodes the latest appearance, called the state, and the sequence of state changes, called transitions. A diffusion transformer uses these representations to predict the next frame. Training has two stages. First, the state encoder, frame predictor, and transition encoder are trained. These components are then frozen, and only the change prediction head is trained in the second stage. The error from this head provides an additional anomaly score alongside the usual frame prediction error. The authors evaluate the method on three benchmark datasets: UCSD Ped2, Avenue, and ShanghaiTech. They also test the learned representations by freezing the encoders and training the change prediction head on different inputs (Table 1), and examine the effects of changing transition information during inference (Table 2). The results show that learned transitions improve performance by 1.25, 0.92, and 0.8 percentage points over raw frame change history. Transition conditioning also improves frame error AUC by 11.53 to 14.9 points on Ped2, a larger gain than on the other two datasets. This suggests that the Ped2 frame predictor relies more heavily on transition information. The authors report a very high video macro AUC of 99.66% on Ped2. Overall, the key idea is that features learned for future frame prediction can also predict the next frame change, providing an additional anomaly signal without requiring iterative generation

Critical strengths

The main strength of this paper is that the authors do not just report anomaly detection results, they also try to understand why the learned transition features are useful. In Section 4.2 (pp. 5 to 6, Table 1), they compare four representations using the same change prediction head. This helps isolate the contribution of the features. Learned transitions perform better than raw frame change history on all three datasets. More importantly, in Section 4.3 (p. 6, Table 2), the authors test what happens when transition information is disturbed. On Ped2, removing the transition features reduces frame error AUC from 91.10% to 76.90%, and replacing them with another clip's transitions reduces it to 76.20%. Reversing the temporal order also hurts change based detection across all three datasets. This gives a better idea of what the model is actually learning. Another interesting point is that better frame prediction does not always mean better anomaly detection, as shown in Section 4.4 (pp. 6 to 7, Tables 3 and 4). Finally, the change only scoring path in Section 3.5 (p. 4, Equation 7) avoids iterative frame generation, which could make inference simpler

Critical weaknesses

The main concern is test set leakage. In Section 4.1 (p. 5) and Appendix B.2 (p. 12), the authors select the score fusion weight (λ) and median filter length using test AUC. The best Ped2 result of 99.66% AUC uses a median filter of length 49 selected on test data (Table 4, p. 7). This makes the reported performance unreliable. The authors should select these settings using validation data and report results with and without smoothing. The study also lacks a simple baseline trained directly to predict frame changes. Comparing this model with the proposed transition encoder would show whether diffusion training actually improves performance (Equation 7, p. 4). The comparison with previous methods is also unfair because the evaluation protocols differ. Table 5 (p. 8) compares results obtained using different AUC definitions and smoothing procedures. As Section 4.1 (p. 5, lines 386 to 388) confirms, existing methods retain their original evaluation settings. Strong baselines such as MA PDM and FPDM should be tested under the same conditions. Statistical reliability is another concern. Table 1 (p. 6) reports small gains of around 0.5 to 1.5% without error bars, while ShanghaiTech results rely on a single run (Table 1 and Appendix B.2). Multiple runs with mean and standard deviation are needed. Finally, the efficiency claims in Section 3.5 (p. 4) lack runtime, memory, throughput, and latency measurements. The authors should also clarify whether the method works online without future frame information. These issues limit reproducibility and raise concerns about real world performance, especially in safety critical surveillance

Questions for authors
  1. Section 4.1 and Appendix B.2 state that scoring parameters, including the fusion weight and median filter length, are selected using test AUC. Could you select these parameters using validation data and report the new test results, particularly whether the 99.66% AUC on Ped2 is maintained?
  2. To isolate the contribution of diffusion training, could you train the same transition encoder directly using the future change prediction loss in Equation 7, without generative pretraining, and compare both models under the same training and evaluation settings?
  3. Table 5 compares the method with existing approaches, but Section 4.1 notes that prior methods retain their original evaluation protocols. Could you re-evaluate key baselines, such as MA-PDM and FPDM, using the same video macro AUC calculation and temporal smoothing as your method to ensure a fair comparison?
  4. Could you report the mean and standard deviation across multiple random seeds for the main results, and measure latency, throughput, and GPU memory for both the change only and frame generation paths? Given the use of video level normalization and temporal filtering, could you also clarify whether the change only path supports causal, online detection?
3 comments

Thank you for highlighting the representation probes, temporal interventions, and change-only path. We completed additional experiments addressing evaluation selection, direct training, seed variability, and computational cost.

  1. Evaluation selection and the Ped2 result

The historical Ped2 predictions reproduce 99.657% macro AUC and 97.977% pooled AUC under the original test-selected post-processing. Without smoothing or whole-video normalization, the same change scores yield 95.755% macro and 92.865% pooled AUC.

We separately conducted controlled retraining with video-disjoint normal training/validation splits, validation-loss checkpoint selection, and one common score without fusion, smoothing, or whole-video normalization. The pretrained explicit-transition model obtains macro AUCs of 93.76 ± 0.19%, 82.60 ± 0.09%, and 78.95 ± 0.06% on Ped2, Avenue, and ShanghaiTech.

We will use the controlled results for the main quantitative discussion and retain the historical result as a post-processing diagnostic. These involve different training runs, and the benchmarks had been examined previously.

  1. Direct training versus generative pretraining

We completed four variants: A: Raw-sequence encoder, directly trained for future-change prediction. B: Raw-sequence encoder, generatively pretrained, then frozen for change-head training. C: Explicit-difference encoder, directly trained for future-change prediction. D: Explicit-difference encoder, generatively pretrained, then frozen for change-head training.

Pooled ROC-AUC (%), mean ± sample SD over seeds 42, 7, and 123:

VariantPed2AvenueShanghaiTech
A89.81 ± 0.2285.30 ± 0.0268.66 ± 0.27
B92.34 ± 0.3782.21 ± 0.1968.88 ± 0.13
C88.14 ± 0.7785.82 ± 0.0968.74 ± 0.12
D91.87 ± 0.4482.42 ± 0.1168.63 ± 0.09

The variants share the change-head architecture, target, loss, and scoring. Direct models receive 10,000 updates; pretrained models receive 5,000 generative plus 5,000 head updates. This matches update count, not compute or convergence.

The paired D-C differences are +3.73, -3.40, and -0.11 percentage points. Pretraining benefits Ped2, direct training is particularly effective on Avenue, and ShanghaiTech is closely matched. We will emphasize this dataset-dependent finding.

  1. Common-protocol external comparison

We evaluated local MA-PDM checkpoints on identical scored-frame cohorts and labels, using the same metrics without smoothing or whole-video normalization.

DatasetModelPooled / macro AUC (%)
AvenueD82.42 / 82.60
AvenueMA-PDM local86.73 / 87.52
ShanghaiTechD68.63 / 78.95
ShanghaiTechMA-PDM local65.49 / 78.74

D summarizes three training seeds; MA-PDM uses one local checkpoint per dataset. Evaluation cohorts and metrics are aligned, while training budgets, preprocessing, and checkpoint selection remain method-specific. These local results are distinct from published MA-PDM numbers. FPDM remains a comparison to complete.

Pooled AUC uses all scored frames; macro AUC averages over videos containing both classes.

  1. Statistical reliability and efficiency

The controlled study contains 36 models: four variants × three datasets × three seeds, including ShanghaiTech.

On an NVIDIA RTX PRO 6000 Blackwell Max-Q, with BF16 autocast and batch size one, decoded-path measurements are:

DatasetPathMedian / P95 msScored frames/s
Ped2Change5.77 / 6.15170.0
Ped2Eight-step generation35.05 / 35.2728.5
AvenueChange5.67 / 5.70176.1
AvenueEight-step generation35.47 / 36.0728.1
ShanghaiTechChange5.59 / 5.61178.9
ShanghaiTechEight-step generation34.37 / 35.3329.0

Peak PyTorch-allocated memory is 0.429 versus 0.927 GiB. Each condition uses five warm-ups and 50 timed calls on normal-validation clips, including decoding, preprocessing, transfers, inference, and score return. Camera/network acquisition is outside scope. Directly trained change models share the efficient generation-free path.

  1. Causal operation

Raw change scoring uses past frames to predict the next change and computes the error when the target frame arrives, without subsequent frames. We distinguish this causal path from historical offline post-processing: whole-video normalization uses video-level information, and the centered 49-frame median filter uses up to 24 future frames. The controlled evaluation uses unfiltered causal scores.

Reviewer_joG1OpenReview ↗

Predictive Transitions for Video Anomaly Detection

Thank you for the detailed response and new experiments. The new results confirm my main concern. The 99.66% AUC on Ped2 depends on post-processing tuned on the test set. Without it, the result drops to 95.76%, and with the new clean protocol it is 93.76%. The main claims should be based on these new numbers.

The direct-training results show that diffusion pretraining is not clearly needed. It helps on Ped2, hurts on Avenue (82.42 vs 85.82), and makes almost no difference on ShanghaiTech. Also, changing only the error aggregation moves Ped2 from 77.63% to 91.87%, so much of the gain comes from the scoring design, not the learned features. Under the same protocol, MA-PDM is still better on Avenue (86.73 vs 82.42), and FPDM is not compared. The seed results, runtime numbers, and the explanation of online use are clear. Overall, the rebuttal is honest, but it shows the original results were too optimistic.

We agree that the main claims should be based on the fixed-protocol results. For the generatively pretrained difference encoder, the revision now reports 91.87% pooled AUC on Ped2 as the primary result (Table 1, p. 6); its video-macro AUC is 93.76%. Section 4.3 distinguishes these three-seed results from the 95.76% macro AUC obtained by removing post-processing from the historical checkpoint. The original 99.66% is retained only to document the effect of test-selected processing. We also report the Avenue comparison explicitly: under the common evaluation protocol, local MA-PDM achieves 86.73% pooled AUC versus 82.42% for our pretrained model (p. 7).

Reviewer_hybDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a future frame prediction-based anomaly detection method for videos. A diffusion transformer is conditioned on a joint input of appearance, ordered frames, and feature changes. Once trained, the model is frozen and a small head is fit on it to predict the next local frame change.

Ped2, Avenue, and ShanghaiTech are used to evaluate the method.

Critical strengths

The paper is well written and explains the approach well enough.

The overall explanation and discussion are thorough, and the idea has some novelty, including the usage of a diffusion model for the purpose of anomaly detection.

Critical weaknesses

I have the following concerns:

  1. The overall novelty of the paper is severely limited. The method proposed future frame prediction, which is not a new idea. While the underlying baseline is different, the overall approach is similar to [1].

  2. The experimental results/comparisons are not fair/appropriate. For example, in Table 5, the paper reports video-macro ROC-AUC for their approach, whereas, to the best of my knowledge, the compared methods report whole-dataset-based ROC-AUC. Video-macro ROC-AUC provides a significant advantage over the conventional approach to AUC calculation.

  3. Did the authors analyze the time cost of using a diffusion model, and compare it with the closest counterpart future frame prediction and other relavant approaches?

  4. The paper lacks comparisons with recent SOTA methods. For example, most of the recent VAD methods now report over 99% performance on ped2.

  5. Expanding on point 2, the comparison between [1] and the proposed approach should be discussed in detail, with both evaluated on the same AUC metric as well as FPS, as the model design is notably similar.

[1] Future Frame Prediction for Anomaly Detection - A New Baseline

Questions for authors

As discussed in Weaknesses section, I have concerns about AUC, novelty, and fair comparisons. Looking forward to the authors' response, and will reconsider my rating accordingly.

4 comments

Thank you for recognizing the clarity of the paper. We completed additional controls to clarify the contribution, evaluation metrics, and computational cost.

  1. Contribution relative to future-frame prediction

Our contribution centers on evaluating transition representations learned through future-frame generation and using them through a generation-free change-prediction readout. Frozen probes examine accessible information, while temporal interventions examine sensitivity to ordered changes.

We additionally compared raw-sequence versus explicit-difference encoders under direct training and generative pretraining. For explicit differences, pretraining changes pooled AUC by +3.73 percentage points on Ped2, -3.40 on Avenue, and -0.11 on ShanghaiTech. These results identify the dataset-dependent role of pretraining: Ped2 benefits from the generative objective, Avenue favors direct change prediction, and ShanghaiTech is closely matched. We will frame the contribution around these measured representation and deployment properties.

  1. AUC definitions and evaluation

Pooled AUC uses all scored frames. Video-macro AUC averages per-video AUCs over videos containing both normal and anomalous scored frames. We will separate these metrics throughout the comparison tables.

The historical Ped2 result is reproduced as 99.657% macro AUC and 97.977% pooled AUC under its test-selected post-processing settings. We distinguish this exploratory result from new controlled training runs using video-disjoint normal training/validation splits, validation-loss checkpoint selection, and a common score without smoothing, whole-video normalization, or fusion.

For the pretrained explicit-transition model, three-seed mean ± sample SD is:

DatasetPooled AUC (%)Macro AUC (%)
Ped291.87 ± 0.4493.76 ± 0.19
Avenue82.42 ± 0.1182.60 ± 0.09
ShanghaiTech68.63 ± 0.0978.95 ± 0.06

These are follow-up training runs on previously examined benchmarks, distinct from rescoring the historical checkpoint. They provide a consistent basis for the main quantitative discussion.

  1. Computational cost

We measured both paths on an NVIDIA RTX PRO 6000 Blackwell Max-Q with BF16 autocast, batch size one, five warm-up calls, and 50 timed calls. Measurements include decoding/preprocessing, transfers, inference, and returning scores; camera/network acquisition is outside the scope.

DatasetChange-only median ms / scored frames/sEight-step generation median ms / scored frames/s
Ped25.77 / 170.035.05 / 28.5
Avenue5.67 / 176.135.47 / 28.1
ShanghaiTech5.59 / 178.934.37 / 29.0

Peak PyTorch-allocated memory is 0.429 GiB versus 0.927 GiB. Generation-free scoring provides approximately sixfold higher throughput in this pipeline. Directly trained change models share this efficiency property.

  1. External comparisons

We completed local MA-PDM evaluations on identical scored-frame cohorts and labels, with matching AUC definitions and no smoothing or whole-video normalization. MA-PDM obtains pooled AUCs of 86.73% on Avenue and 65.49% on ShanghaiTech, compared with 82.42 ± 0.11% and 68.63 ± 0.09% for our pretrained explicit-transition model.

These are single local MA-PDM checkpoints, with method-specific training and preprocessing. We will distinguish them from published benchmark results and identify the comparison as evaluation-aligned rather than training-matched.

  1. Comparison with Future Frame Prediction [1]

The methodological distinction is the separately evaluated transition representation and generation-free readout. A matched reproduction of [1] reporting AUC and FPS remains to be completed. Current speed results compare our two inference paths; the MA-PDM experiments provide an external comparison under aligned evaluation metrics.

Together, the additions clarify the contribution and provide explicit, reproducible comparisons of representation quality and inference cost.

Dear Authors,

Thank you for your response.

As I mentioned in my earlier comments, and as the new experiments now show, Macro has an advantage over conventional evaluation. The results/discussions in the manuscript need significant revisions. I currently do not see any change in the uploaded PDF of the manuscript, and therefore, keep my origional rating.

Thank you for pointing out that our rebuttal was not yet reflected in the uploaded PDF. We have now uploaded the revision. Table 1 (p. 6) reports pooled frame-level ROC-AUC as the primary metric; video-macro AUC is reported separately. We removed the former Table 5, which placed our test-selected macro result alongside published results obtained under different protocols. Section 4.4 (p. 7) now compares our method with local MA-PDM checkpoints on the same scored frames and reports measured inference cost. We do not claim a matched AUC or FPS advantage over FFP.

Thank you, dear authors, for your efforts and responses.

I will read the revised manuscript and will update my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.