PULSE: Prototype-Based Latent Surprise Detects Physics Anomalies

Physics anomalies are events in which a system stops following its usual physical dynamics. Such anomalies appear normal in a single frame and only become apparent through the temporal evolution of the scene. One approach is to use a world model to predict how a scene evolves and measure surprise when the system deviates from the prediction. However, such predictive approaches can assign high surprise to any unexpected change, whether or not the change is a deviation from normal dynamics. We propose PULSE, which learns a system’s normal dynamics by training a model on normal videos in the feature space of a frozen encoder and scores frames using the resulting latent surprise. Feature-space world models typically regress future features, but when several futures are normal, regression can average them into a single intermediate representation, causing normal outcomes to receive high surprise scores. PULSE instead clusters normal features into prototypes and assigns a probability to each. These prototypes provide explicit normal references for anomaly scoring, while the predicted probabilities preserve distinct plausible futures. PULSE sets a new state of the art with the highest average AUROC on three physics anomaly benchmarks that cover real-world object interactions, industrial processes and video games, achieving 98.6 on Phys-AD, 76.5 on IPAD and 72.8 on TempGlitch. Our ablations show that using prototype-based prediction provides the largest methodological improvement at +6.3 AUROC points.