acceptodds
Under review as a conference paper at ICLR 2027

PULSE: Prototype-Based Latent Surprise Detects Physics Anomalies

Abstract

Physics anomalies are events in which a system stops following its usual physical dynamics. Such anomalies appear normal in a single frame and only become apparent through the temporal evolution of the scene. One approach is to use a world model to predict how a scene evolves and measure surprise when the system deviates from the prediction. However, such predictive approaches can assign high surprise to any unexpected change, whether or not the change is a deviation from normal dynamics. We propose PULSE, which learns a system’s normal dynamics by training a model on normal videos in the feature space of a frozen encoder and scores frames using the resulting latent surprise. Feature-space world models typically regress future features, but when several futures are normal, regression can average them into a single intermediate representation, causing normal outcomes to receive high surprise scores. PULSE instead clusters normal features into prototypes and assigns a probability to each. These prototypes provide explicit normal references for anomaly scoring, while the predicted probabilities preserve distinct plausible futures. PULSE sets a new state of the art with the highest average AUROC on three physics anomaly benchmarks that cover real-world object interactions, industrial processes and video games, achieving 98.6 on Phys-AD, 76.5 on IPAD and 72.8 on TempGlitch. Our ablations show that using prototype-based prediction provides the largest methodological improvement at +6.3 AUROC points.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
video anomaly detection, physics anomalies, unsupervised learning, world models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_c3feOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PULSE, a prototype-based latent prediction framework for anomaly detection in videos. The method aims to address the limitations of conventional feature regression, which may average multiple plausible future states and consequently assign high anomaly scores to normal outcomes. Specifically, PULSE extracts visual features using a frozen encoder, then clusters normal features into prototypes, and trains a lightweight predictor to estimate the probability distribution over future prototype assignments. Anomalies are identified through latent surprise, measured by the negative log-probability of the observed prototype. Experiments on three physics anomaly detection benchmarks, Phys-AD, IPAD, and TempGlitch, show competitive performance against existing approaches.

Critical strengths
  • The paper presents a simple yet effective framework for physics anomaly detection by learning system-specific normal dynamics from videos. The use of prototype-based latent prediction provides an intuitive alternative to conventional feature regression, potentially alleviating the ambiguity caused by multiple plausible future states without requiring explicit physical rules or anomaly annotations.

  • The proposed method achieves competitive results on three physics anomaly detection benchmarks, covering industrial processes, mechanical interactions, and gameplay environments, and outperforms existing zero-shot, supervised and unsupervised baselines.

Critical weaknesses

The paper motivates prototype-based prediction as a way to address the mode-averaging problem in continuous feature regression. However, it remains unclear whether the proposed method genuinely captures multiple plausible future states, as the predicted prototype distribution is still derived from a single continuous feature representation.

Questions for authors
  • The introduction transitions abruptly from the limitations of pixel-space prediction to feature-space prediction and subsequently to prototype-based modeling, without clearly establishing the connections between these design choices. Could the authors further clarify the practical significance of the problem and provide a more coherent motivation for the proposed framework?

  • The proposed method learns normal dynamics from training videos of a particular system. Does this require training a separate predictor and constructing a new prototype vocabulary for every target environment? If so, how practical is this framework for real-world applications involving diverse or previously unseen physical systems? An evaluation of cross-system generalization would help establish its broader applicability.

Reviewer_wdsTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PULSE, an unsupervised method of detecting anomalies in videos. The method uses DINOv2 ViT-S/14 to extract patch-level features from normal video frames and clusters them into prototypes, which serve as reference patterns. Then, a transformer-based predictor learns which prototypes are more likely to appear in the future frames. Predicted and observed features are then mapped to a distribution over prototypes. Afterwards, the latent surprise of an observed patch is computed as the negative log-probability that the predictor assigned to its prototype, meaning that a patch receives a higher surprise score when its prototype was unlikely based on the current context. These scores are averaged to obtain frame-level and video-level anomaly scores. The method obtains state-of-the-art results on three physics anomaly detection benchmarks: Phys-AD, IPAD, and TempGlitch, achieving average AUROC scores of 98.6, 76.5, and 72.8, respectively.

Critical strengths
  1. Strong performance across multiple settings: PULSE achieves the highest AUROC on all three reported benchmarks, covering industrial processes, gameplay, and phisical objects.
  2. The simplicity of the method: PULSE combines a frozen encoder (DINOv2 ViT-S/14) with a small world model that predicts future prototypes, requiring only normal videos for training.
  3. Prototype-based prediction helps: the ablations show a 6.3-point AUROC improvement over feature regression on IPAD, while also examining several design choices, such as prototype count and temporal context.
Critical weaknesses
  1. Limited validation of the main motivation: although prototype prediction improves performance over feature regression, the paper does not directly demonstrate that it preserves multiple plausible futures. One way of validating this hypothesis would be trying an experiment with multiple valid outcomes for the same context, such as the paper’s example of a ball that may roll either left or right.
  2. The importance of frame order is not evaluated: it is unclear how much the method relies on motion rather than appearance. Testing on shuffled context frames would help clarify this.
  3. Lack of multi-seed evaluation: the authors report results using a single random seed, making it difficult to assess the stability of the results.
  4. Unclear hyperparameter selection: the paper lists the selected hyperparameters, but does not explain how they were selected.
Questions for authors

Please see the weaknesses above.

Reviewer_DUNkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce PULSE, a method for detection abnormal physical behavior in videos by learning only from normal videos. A frozen visual encoder extracts features that are clustered into prototypes and trains a model to predict probabilities over future prototypes. An observed outcome gets a high anomaly score when is assigned a prototype that was unlikely based on the preceding frames.

Main contribution is the prototype-based future prediction objective coupled with an anomaly score defined by the probability of the observed prototype assignment.

Critical strengths
  • Modular architecture: a frozen visual encoder, fixed prototypes and a small predictor.
  • Training requires only normal examples. This gives an advantage for scenarios where anomalies are hard to collect and annotate.
  • Useful ablation studies
Critical weaknesses
  • Averaging frame-level scores over the entire video may dilute the signal from short anomalies, especially in longer videos.
  • Authors attribute improvements to better handling of multiple plausible futures. While the main ablation introduces clustering which improves AUROC, it does not directly demonstrate the gain comes the better handling multiple plausible futures.
  • Results are reported with a single seed. Multiple runs would help distinguish meaningful improvements from variation, particularly for the smaller differences.
Questions for authors

Could you provide evidence that the improvement comes from better handling of multiple plausible futures? When computing video-level anomaly scores, how does averaging frame-level scores over the entire video affect the detection of short anomalies, especially in longer videos? Could you provide more details for the TempGlitch data split? Did you keep the clips from the same source within the same split?

Reviewer_V2jUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PULSE is an unsupervised physics anomaly detection method that learns the normal dynamics of a system using only normal images. The authors argue that a world model that regresses future features averages multiple normal futures, resulting in high surprise even for normal results. Instead, fixed DINOv2 ViT-S/14 patch features are clustered into K=32,768 prototypes using k-means, and a 4-block transformer predicts the prototype distribution of the next 2 frames from the contextual 4 frames. Learning is performed by the cross-entropy of the predicted distributions and the centered and sharpened observational soft assignments, while inference is performed by spatially and temporally averaging the negative logarithm of the predicted probability of the observational hard assignments as the patch-level surprise. It claims the highest average of the three benchmarks with Phys-AD video-level AUROC 98.6 (supervised learning MT-PIVLM exceeding 96.7), TempGlitch 72.8 (zero-shot JEPA IntPhys 63.9), and IPAD frame-level 76.5 (INFP 71.8), emphasizing that it surpasses 7–33B VLM with 7.4M trainable parameters. The baseline marked with † was evaluated and trained directly by the authors using open code, the TempGlitch split was author-defined, and all experiments were single-seeded. In IPAD ablation, the prototype objective function is +6.3 compared to ℓ1 regression and +5.3 for K 512→32,768, and the influence of the predictor structure is within ±1.3.

Critical strengths

It demonstrates consistent average performance superiority across different types of benchmarks. It reports the highest average AUROC among comparison methods in Object Interaction (Phys-AD), Industrial Equipment (IPAD), and Game Video (TempGlitch); notably, in the frame-level evaluation of IPAD, it is 4.7 points higher than the dedicated design INFP, and in TempGlitch, it is 8.9 points higher than the strongest baseline. In Phys-AD, it is the best in 20 out of 22 categories (including ties) and surpasses the supervised learning MT-PIVLM. This is achieved with 7.4M (total 41.9M) trainable parameters. The fact that a single unsupervised design surpasses both the dedicated design and the billion-parameter VLM across all three domains is the primary rationale supporting its adoption.

The selection of the core design is supported by direct ablation. In the comparison in Table 4a/12, where only the loss was changed under the same encoder, K, and training settings, the prototype-based CE improves from 70.2 to 76.5 compared to the ℓ1 regression. By providing experiments on the number of prototypes (+5.3 from 512 to 32,768), context and prediction length, encoder size, and predictor depth and attention type (within ±1.3), the author's interpretation that improvement stems from prototype-based goals rather than the predictor structure is numerically supported. This implies that the method's core argument can be separated into ablation, and is also significant in that the additional control experiments required in the weaknesses section below can be conducted within the same framework.

Critical weaknesses
  1. It was not separated whether performance improvement stems from conditional temporal prediction or from the finer partitioning of the normal feature space, and the multiple futures hypothesis was not directly tested. (a) MemAE and DMAD are included, but there are no non-predictive controls using the same frozen encoder and codebook—such as −log of prototype prior frequencies, nearest prototype distance, normal patch kNN, and predictions that copy the assignment of the last context frame. The fact that the majority category of Phys-AD is 100.0 suggests that separation often occurs based solely on the appearance of fixed features; therefore, without this control, “latent surprise” cannot be distinguished from appearance outlier detection. The context length ablation in Table 4c partially supports the usefulness of temporal information but does not replace the comparison with the absence of a predictor. (b) The +6.3 relative to ℓ1 regression supports changes in the objective function and scoring method, but does not show that the regression failure occurred in the multiple futures. Moreover, since the optimal value of ℓ1 is the median rather than the conditional mean, this ablation does not verify the hypothesis in the introduction that "regression averages the future" (corresponding to L2 regression). Additionally, since the prediction distribution of PULSE is the softmax of the distance from a single prediction feature to the prototype, the prediction point must lie between two separated normal futures to simultaneously give high probabilities, and the expressible conditional distribution is structurally limited. It is necessary to compare the probability of each future with the false alarms of the normal future in a control environment where multiple normal futures are possible within the same context, and to extend the objective function ablation to Phys-AD·TempGlitch. If the non-predictive control in (a) is not presented in the rebuttal, it is difficult to acknowledge the attribution of the core contribution, and in that case, the evaluation will be lowered.

  2. The uncertainty of the results and baseline equivalence were not sufficiently specified. The entire experiment was a single run of seed 123, and there is no repeated variance. Since TempGlitch uses an author-defined 80/20 split, split sensitivity and whether clips from the same game or scene are split between train and test are important. Comparisons with JEPA IntPhys involve the difference between zero-shot and normal data adaptation, so improvements to the method itself must be interpreted in terms of the trained baseline and control ablation; however, since DINO-Foresight (†), the strongest training baseline, was trained directly by the authors, it is not specified whether the resolution, fps, epoch, and learning rate were equivalent to PULSE. To isolate the contribution of the discrete prediction itself, in addition to the current ablation that replaces prototype CE with continuous feature regression in the same predictor, a variation is needed that overlays PULSE's codebook and CE objectives on top of the DINO-Foresight predictor. These are fixable flaws, but currently, it is not possible to determine whether the reported gap (IPAD +4.7, TempGlitch +8.9) exceeds the differences in seed, split, and tuning.

Questions for authors
  1. Can non-predicted scores—−log of prototype prior frequencies, nearest prototype distance, and normal patch kNN—and predictions that copy context removal and the assignment of the last context frame be compared across the three benchmarks using the same DINOv2 feature, codebook, and aggregation methods? What contribution does the conditional future prediction leave after these are excluded?

  2. In a controlled environment where multiple normal futures are possible within the same context, can the probabilities given by L1 regression (and L2 regression) and PULSE for each future be compared with the false alarms for normal futures? Please also analyze whether Eq. (3), the distance softmax on a single prediction feature, can simultaneously provide high probabilities for two separate modes.

  3. With what settings (encoder, resolution, frame sampling, epoch, learning rate, tuning criteria) was DINO-Foresight trained and evaluated? If there is a variant that predicts discrete codes as CE, was it included? If not, can a variant that overlays PULSE's codebook and CE objectives on the same predictor be evaluated? What accounts for the 29-point difference between the IPAD's DINO-Foresight 41.0, which performs continuous regression on the same DINOv2 features, and the identical L1 ablation setting of 70.2?

  4. Can you provide confidence intervals for the AUROC difference and results obtained with at least 3 seedings for key comparisons? What are the TempGlitch split list, the train/test split criteria for the same game, session, and source video, and the sensitivity of split resampling?

  5. Hard assignments for observation features far from all prototypes fall onto a specific prototype; however, if that prototype receives a high probability in the prediction distribution, the surprise value decreases. How often does this actually occur? Please analyze the complementarity between the assignment NLL and codebook distance, and the prototype share distribution after centering.

  6. How does the detection of local and short-duration anomalies (shooting errors, S06–S08) change if TempGlitch's video time aggregation and the IPAD's spatial and overlapping clip aggregation are switched from mean to max/top-k?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.