- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a state–transition framework for normal-only video anomaly detection. The observed video history is decomposed into a state representation, which captures the current appearance, and a transition representation, which encodes ordered frame and feature differences. These representations jointly condition a diffusion Transformer trained for future-frame generation. After the generative model is trained, the encoders are frozen and a lightweight change-prediction head is trained on the transition representation to predict the next local frame change. The resulting change-prediction error is used as an anomaly score, either alone or in combination with the conventional future-frame prediction error.
The paper evaluates the learned transition representation using frozen-representation probes and inference-time interventions. The experiments compare transition features with state features, feature differences, and raw frame-change history, and further examine the effects of perturbing transition content and temporal order. The method is evaluated on Ped2, Avenue, and ShanghaiTech.
Critical strengths
-
The paper investigates transition representations separately from conventional frame-prediction error. An interesting aspect of the paper is that it does not evaluate transition information only through its contribution to future-frame reconstruction/prediction. Instead, the authors directly probe the learned transition representation through a future-change prediction task. This helps distinguish whether temporal information is present in the representation even when it does not improve frame-error-based anomaly detection.
-
The representation-probing experiments are useful and reasonably controlled. Table 1 applies the same change-head architecture to several frozen representations, including the current state, the last feature difference, raw frame-change history, and the learned transition representation. Learned transition features consistently outperform raw frame-change history across all three datasets, although the margins are relatively modest.
-
The temporal-order interventions provide some evidence that the learned representation captures ordered temporal information. Reversing or cyclically shifting the transition history reduces change-based anomaly detection performance, especially on Ped2 and Avenue. This is a useful diagnostic experiment showing that the transition encoder does not merely encode the overall magnitude of motion/change.
Critical weaknesses
-
The paper does not establish that explicitly separating a transition representation provides a meaningful advantage over conventional multi-frame temporal modeling. Existing future-frame prediction methods already take multiple frames as input, and sufficiently expressive temporal encoders can implicitly learn motion and temporal dynamics. The key missing comparison is therefore a capacity-matched model that directly encodes the same sequence of raw frames without explicitly constructing frame/feature differences. Table 1 only compares the proposed transition representation with state features, feature differences, and raw frame-change history. As a result, the paper shows that its learned transition feature is better than raw differences, but does not demonstrate that explicit transition modeling itself is necessary or superior to standard multi-frame encoding.
-
The necessity of the generative training framework is unclear. The transition encoder is first trained through conditional future-frame generation, then frozen, and finally used for future-change prediction. However, there is no direct baseline where the transition encoder and change head are trained end-to-end for next-change prediction. If such a simpler model achieves similar performance, the diffusion Transformer, flow-matching objective, and two-stage training procedure may not be essential. The paper itself acknowledges that the necessity of generative training remains unresolved.
-
The final reported performance is difficult to attribute to the proposed transition representation because of substantial dataset-specific tuning and additional score engineering. The score transformation, frame/change mixing weight , and median-filter length are selected using test AUC, and the selected values differ substantially across datasets. In addition, the change target and anomaly score involve several designed components, including signed pooling, magnitude components, active/static loss balancing, and top- aggregation. Without sufficient ablations, it is difficult to determine how much of the gain comes from the transition representation itself versus these target, scoring, and post-processing choices.
-
Several experiments needed to support the central claims are missing or insufficiently robust. The method always uses six observed frames, despite the claim that ordered transition history is important, so an ablation over history length would be valuable. The benefit of transition conditioning is also inconsistent across datasets: State + Transition improves over State on Ped2, but is worse on Avenue and nearly identical on ShanghaiTech. Moreover, ShanghaiTech uses only one backbone seed even though several reported differences are below one percentage point. Finally, the comparison with published methods is not strictly controlled because the proposed method reports video-macro AUC while prior results retain their original evaluation protocols.
Questions for authors
-
Can the authors provide a capacity-matched baseline that directly encodes the same sequence of raw frames without explicitly constructing frame or feature differences? This comparison seems necessary to establish whether explicit transition modeling provides an advantage over conventional multi-frame temporal encoding.
-
How does the proposed transition encoder compare with a standard temporal encoder, such as a Transformer operating directly on per-frame features, under the same architecture size and training objective?
-
How does the proposed generatively pretrained transition encoder compare with one trained directly and end-to-end for future-change prediction? This would help determine whether future-frame generation is necessary for learning the useful transition representation.
-
What happens if the diffusion/flow-matching pretraining stage is removed entirely and only the change-prediction objective is used?
-
Why are the score transformation, , and median-filter length selected based on test AUC? Can the authors report results using only validation-selected hyperparameters?
-
Can the authors report a single common scoring/post-processing configuration across all three datasets? This would help assess how much performance depends on dataset-specific tuning.
-
How sensitive are the final results to and the median-filter length? A sensitivity plot or table would help clarify whether the reported values correspond to broad stable regions or narrow optima.
-
Can the authors provide ablations for the future-change target design? In particular, what are the individual contributions of positive/negative signed pooling and the average/max magnitude components?
-
How much does active/static cell loss balancing contribute to the change-prediction performance?
-
How much does the top- aggregation contribute compared with simply averaging all local errors?
-
How does performance change with the number of observed frames? For example, results for 2, 4, 6, and 8 frames would help determine whether modeling a longer ordered transition history is actually beneficial.
-
How does the method compare when only the most recent transition is used instead of the full transition history? This would help separate the benefit of ordered history from the benefit of simply using frame differences.
-
Why does State + Transition conditioning underperform State-only conditioning on Avenue? Does this indicate that the transition condition may sometimes interfere with the future-frame generator?
-
Can the authors explain why transition conditioning contributes strongly on Ped2 but very little on Avenue and ShanghaiTech? Is this related to scene complexity, motion statistics, or optimization?
-
Can the authors report multiple-seed results on ShanghaiTech? Several differences involving the transition representation are below one percentage point.
-
Can the authors evaluate the proposed method and key baselines under exactly the same AUC protocol? Since Table 5 combines the paper's video-macro AUC with previously published numbers using their original protocols, it is difficult to interpret the comparison quantitatively.
-
Can the authors clarify the intended primary contribution of the paper? Is the main contribution the state–transition generative architecture, the learned transition representation, or the change-based anomaly scoring mechanism? The current experiments provide different levels of support for these components.
-
Why is the evaluation limited to Ped2, Avenue, and ShanghaiTech? Could the authors provide results on more recent and diverse VAD benchmarks, such as UBnormal or other modern datasets? Given that the proposed method explicitly models temporal transitions, evaluation on datasets with more diverse scenes, anomaly types, and motion patterns would help assess whether the observed benefits generalize beyond the traditional benchmarks used in this paper.
4 comments
Remaining Concerns After Rebuttal
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my questions, especially the comparisons between raw and difference-based encoding and between direct training and generative pretraining.
After reviewing the new results, I still have two main concerns.
1. Performance and evaluation protocol.
The original results appear to depend heavily on test-selected post-processing. For example, the Ped2 macro AUC drops from 99.657% to 95.755% without smoothing or whole-video normalization. The new fixed-protocol results are also substantially lower than the originally reported numbers, although I understand these come from separate training runs.
The aggregation ablation is also concerning. On Ped2, changing from mean local error to mean + top-four aggregation improves AUC from 77.63% to 91.87%. This suggests that a substantial part of the reported performance depends on the scoring design rather than the learned representation alone. While the revised evaluation protocol is an improvement, I am still not convinced that the original performance claims are sufficiently supported.
2. Motivation for the proposed framework.
The new A–D experiments are helpful, but they also raise questions about the necessity of the proposed design. Explicit difference encoding does not consistently outperform raw temporal encoding. Similarly, generative pretraining improves performance on Ped2 but performs considerably worse than direct change-prediction training on Avenue, with almost no difference on ShanghaiTech.
I do not expect the proposed method to outperform every baseline on every dataset. However, if raw temporal encoding performs similarly and direct change prediction can achieve comparable or better results, it is unclear what advantage the proposed state–transition generative framework provides over these simpler alternatives.
Overall, I appreciate the additional experiments and clarifications. They address several of my original questions, but I do not think they resolve the two main concerns above. I therefore maintain my original rating.
Your distinction between finding useful information in a representation and showing that the proposed framework is needed to learn it helped us revise the paper. On evaluation, Tables 1–2 (p. 6) now show both aggregation rules on the same checkpoints. For example, on Ped2 the pretraining gain over direct training with difference inputs is +3.73 points with mean-plus-top-four aggregation but only +0.72 points with mean alone. We therefore attribute the larger gain to the representation and its readout together, not to the representation alone.
On motivation, the contribution we retain is that a temporal representation learned to condition frame generation can also score future changes without running the generator at inference. This is a useful property of an already trained generative model, not a reason by itself to prefer generative training when change detection is the sole objective. The revised conclusion (p. 7) makes that distinction explicit: pretraining helps on Ped2 under the primary scoring rule, direct training is stronger on Avenue, and the difference is small on ShanghaiTech.
Official Comment by Reviewer ooEd
Thank you for the detailed response and revisions. However, my main concerns remain unresolved.
The revised results raise further questions about the novelty and effectiveness of the proposed framework. If raw temporal encoding achieves comparable performance to the proposed difference-based transition representation, and direct training performs similarly to or better than generative pretraining, it is unclear what advantages the proposed state–transition framework provides over these simpler alternatives.
Furthermore, the detection performance appears to depend heavily on the aggregation strategy. Without the top-four aggregation, the performance drops substantially, and the mean-only results do not provide convincing evidence of an advantage over existing methods. I also noticed that the revised paper no longer includes the original comparison table with existing VAD methods. While I understand that differences in training and evaluation protocols make direct comparisons difficult, I believe that comparisons under a consistent evaluation protocol, or at least clearly qualified comparisons with existing methods, are important for assessing the effectiveness of the proposed approach.
Overall, although the revisions clarify the limitations of the method, I am still not convinced that the proposed state–transition framework offers a sufficiently novel or effective approach to video anomaly detection. I therefore maintain my original rating.
Thank you for the detailed questions. We completed controls separating representation, training objective, aggregation, and evaluation protocol.
Raw-sequence and explicit-difference encoders share widths, temporal/spatial blocks, and the change head. Parameter counts differ by 256 parameters. Raw inputs have six positions and differences have five; capacity is closely matched, while FLOPs differ.
A: Raw/direct change prediction. B: Raw/generative pretraining, then frozen-encoder change-head training. C: Differences/direct change prediction. D: Differences/generative pretraining, then frozen-encoder change-head training.
Pooled AUC (%), mean ± sample SD over three training seeds:
D-B compares encodings under pretraining: -0.48, +0.21, and -0.25 percentage points. D-C compares pretraining with direct training: +3.73, -3.40, and -0.11 points. Encoding performance is closely related; pretraining benefits Ped2, while direct training performs strongly on Avenue.
Direct models receive 10,000 updates; pretrained models receive 5,000 generative plus 5,000 head updates. Targets, head architecture, loss, and scoring are shared. This matches updates, not compute or convergence.
The new study uses video-disjoint normal train/validation splits, validation-loss checkpoint selection, and a common score without fusion, smoothing, or whole-video normalization. Pooled AUC uses all scored frames; macro AUC averages videos containing both classes.
Historical Ped2 macro AUC reproduces 99.657% under test-selected post-processing. New D runs obtain 93.76 ± 0.19% under the fixed protocol. These are separate training runs on previously examined benchmarks.
A retrospective sensitivity audit covers 196 configurations per dataset, varying fusion, median windows, normalization, and filter direction. Historical Ped2 change scores yield 95.755% macro and 92.865% pooled AUC without smoothing or whole-video normalization. This diagnostic is separate from validation-based selection.
For the same D models, three-seed mean pooled AUC is:
Aggregation contributes substantially, particularly on Ped2, and is shared across A-D. Component-wise retraining ablations for signed pooling, average/max magnitude, and active/static loss balancing remain to be completed.
Order interventions demonstrate temporal-order sensitivity; the last-feature-difference probe provides a short-history reference. Matched retraining for 2/4/6/8 frames and a single-transition encoder remains to be completed.
Conditioning interventions characterize dataset-specific behavior. Scene complexity, motion statistics, and optimization remain candidate explanations rather than isolated causes.
The A-D study includes three training seeds on every dataset, including ShanghaiTech.
Local MA-PDM evaluations use identical scored-frame cohorts, labels, and metrics without smoothing or whole-video normalization. Its pooled/macro AUC is 86.73/87.52% on Avenue and 65.49/78.74% on ShanghaiTech. Each uses one local checkpoint with method-specific training and preprocessing, distinct from published benchmark results. Matched FFP and FPDM comparisons remain to be completed.
Earlier normal-only UBnormal experiments cover three backbone seeds, without smoothing or test-video normalization. Transition pooled AUC is 55.66 ± 0.02%, versus 54.89 ± 0.17% for actual-state features and 56.18% for raw change history. These extend the frozen-probe analysis and are distinct from the new A-D protocol.
Our contribution separates temporal information in learned representations, its utility for change-based detection, and generation-free inference cost. The controls clarify the dataset-dependent roles of encoding, training, and aggregation.