How Models Memorize vs. Generalize: Intervening via Features and Labels

Although deep learning is widely adopted for its capability to fit training data effectively, it often memorizes outliers and/or mislabeled instances, a phenomenon known as label memorization. Despite this, there has been no clear distinction between memorization and generalization at the feature level, and how they interact with label memorization. Hence, in this paper, first, we precisely distinguish feature memorization from feature generalization and define the conditions under which they occur. Second, we investigate the interactions among feature memorization, feature generalization, and label memorization, revealing that label memorization suppresses feature memorization while causing feature generalization. These findings offer a systematic understanding of memorization and generalization in deep neural networks.