What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder with a fixed decoder and an encoder trained by full-batch gradient descent, we show that, under suitable regularity conditions, the IM effect emerges over a range of training iterations. Our results characterize when such an interval exists and which outliers can be separated from inliers throughout the interval. Further, we characterize how data distribution and weight initialization affect the strength and persistence of the IM effect, and provide empirical support for these theoretical predictions through simulation studies. Guided by these insights, we study compact representations and EMA-based initialization as two practical strategies for enhancing the IM effect. Experiments on real-world data further evaluate their individual and combined effects on the performance of IM-based outlier detection methods.